From 9260f8ee0a9efe597064b0aa24fb5d345a13a5a0 Mon Sep 17 00:00:00 2001 From: Nehanth Date: Fri, 16 Oct 2026 17:02:09 -0400 Subject: [PATCH 1/3] Longer context by default for the 27B and the 35B MoE: a context ladder The 27B (64K/32K/16K/8K) and the 35B MoE (128K/64K/32K/16K/8K) open at the longest length the room holds, past their old default (16K / 32K) only at full speed: no expert offloaded and no more devices holding layers than at the default. The picker's context row offers 8K/16K/32K/64K (27B) and 8K/32K/64K/128K (MoE). The room page (picker and Start), the room node (pooled host, serve, the OpenClaw plugin) and pooled host's screen all call roomCtxPlan. ?ctx= stays an exact ask. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MkVLt1BijYxEeNWsgihPdN --- README.md | 4 +- cli/README.md | 2 +- cli/lib/hostrun.js | 2 +- cli/lib/hostui.js | 22 +++- cli/lib/lend.js | 6 +- cli/lib/lendrun.js | 4 +- cli/test/hostui_test.mjs | 40 +++++-- docs-site/src/content/docs/code/agent.mdx | 2 +- .../src/content/docs/contributing/models.mdx | 15 +-- .../src/content/docs/internals/sessions.md | 2 +- .../src/content/docs/kits/coding-tools.mdx | 4 +- docs-site/src/content/docs/kits/terminal.mdx | 2 +- docs-site/src/content/docs/models.mdx | 25 ++++- .../src/content/docs/reference/models.mdx | 6 +- .../content/docs/reference/url-options.mdx | 10 +- docs-site/src/content/docs/serve/index.mdx | 2 +- docs-site/src/content/docs/serve/limits.md | 6 +- .../src/content/docs/serve/recipes/index.mdx | 14 ++- docs-site/src/content/docs/terminal.mdx | 6 +- .../src/content/docs/troubleshooting.mdx | 2 +- docs/models.md | 4 +- packages/openclaw/src/models.js | 37 +++--- packages/openclaw/test/plugin_test.mjs | 4 +- packages/openclaw/test/v1_test.mjs | 3 +- packages/room-node/index.js | 3 +- packages/room-node/roomnode.js | 23 ++-- packages/room-node/test/roomnode_test.mjs | 14 ++- room.js | 26 +++-- room/ctxpick.js | 29 +++-- room/models.js | 57 ++++++++-- tests/test_moe.js | 3 +- tests/unit/ctxpick_test.js | 106 ++++++++++++++++++ tests/unit/flash_room_test.js | 2 +- tests/unit/models_test.js | 18 ++- 34 files changed, 377 insertions(+), 128 deletions(-) diff --git a/README.md b/README.md index 2a987fa3..1fca057f 100644 --- a/README.md +++ b/README.md @@ -43,8 +43,8 @@ Models in the room today: | Model | Size | Notes | |---|---|---| -| Qwen 3.8 27B | Q4_0, needs ~17 GB across the room | hybrid Gated DeltaNet + attention, built-in draft layer for speculative decoding, 16K context (up to 64K) | -| Qwen 3.6 35B MoE | Q4_0, needs ~22.5 GB across the room | 256 experts, 8 active per token, so it decodes several times faster than the 27B; 32K context (up to 128K) | +| Qwen 3.8 27B | Q4_0, needs ~17 GB across the room | hybrid Gated DeltaNet + attention, built-in draft layer for speculative decoding; opens at the longest context the room holds at full speed, up to 64K (16K at ~17 GB, 64K at ~20 GB) | +| Qwen 3.6 35B MoE | Q4_0, needs ~22.5 GB across the room | 256 experts, 8 active per token, so it decodes several times faster than the 27B; opens at the longest context the room holds at full speed, up to 128K (32K at ~22.5 GB, 128K at ~23.5 GB) | | Qwen3 1.7B | Q8_0, needs ~5.6 GB across the room for 16K context, ~4 GB for 8K | small and quick, for rooms of phones and light laptops; 16K context, 8K when the room is short of memory | | Qwen3.5 122B MoE | Q4_0, a 72.5 GB download, needs ~73 GB across the room | 256 experts, 8 active per token; in `pooled pull` / `pooled host` (CLI 0.4.0+), on the room page only with `?dev=1`. A Mac Studio plus an RTX 5070 PC keeping its experts in RAM runs it at ~17 tok/s | | Qwen3.8 Flash (**experimental**) | Q4_0, 125B (6B active), ~99 GB on disk | 512 experts, 10 active per token; keeps most of itself on disk, so it needs far less GPU memory than its size; 64K context. See below | diff --git a/cli/README.md b/cli/README.md index 0fe6b1ff..19305aff 100644 --- a/cli/README.md +++ b/cli/README.md @@ -179,7 +179,7 @@ ANTHROPIC_BASE_URL=http://127.0.0.1:8080 ANTHROPIC_API_KEY=pooled \ CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --model pooled ``` -`CLAUDE_CODE_MAX_CONTEXT_TOKENS` is the room's context (the banner prints this line for your room): Claude Code does not know a model called `pooled`, and without it never compacts before the room refuses a prompt. `CLAUDE_CODE_MAX_OUTPUT_TOKENS` (a quarter of the context, at most 32000): Claude Code keeps its output limit (32000) free in the window, so on a room under about 32 k without it, it refuses its own prompt ("Prompt is too long") before sending anything. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` keeps its side requests (titles) from interleaving with the agent's and costing a re-prefill. Its prompt is 15-20 k tokens before any history, so use a room with a long context: the 35B MoE (32 k by default, up to 64 k with `?ctx=65536`) or the 27B (up to 32 k). The 1.7B's 16 k leaves about 1 k tokens after Claude Code's prompt: a file read can already overflow it, and the 1.7B cannot write the compaction summary, so use a bigger model for Claude Code. Each step after the first reuses the room's caches and prefills only the new tool results. MCP servers are fine: up to 1024 tools per request. +`CLAUDE_CODE_MAX_CONTEXT_TOKENS` is the room's context (the banner prints this line for your room): Claude Code does not know a model called `pooled`, and without it never compacts before the room refuses a prompt. `CLAUDE_CODE_MAX_OUTPUT_TOKENS` (a quarter of the context, at most 32000): Claude Code keeps its output limit (32000) free in the window, so on a room under about 32 k without it, it refuses its own prompt ("Prompt is too long") before sending anything. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` keeps its side requests (titles) from interleaving with the agent's and costing a re-prefill. Its prompt is 15-20 k tokens before any history, so use a room with a long context: the 35B MoE or the 27B, which a room opens at the longest context it holds at full speed (up to 128 k and 64 k; at least 32 k and 16 k when the room holds those; `?ctx=` or `--ctx` asks for an exact length). The 1.7B's 16 k leaves about 1 k tokens after Claude Code's prompt: a file read can already overflow it, and the 1.7B cannot write the compaction summary, so use a bigger model for Claude Code. Each step after the first reuses the room's caches and prefills only the new tool results. MCP servers are fine: up to 1024 tools per request. ### opencode diff --git a/cli/lib/hostrun.js b/cli/lib/hostrun.js index 6cc61d58..f70f2b3e 100644 --- a/cli/lib/hostrun.js +++ b/cli/lib/hostrun.js @@ -19,7 +19,7 @@ export async function runHostInteractive(opts, { prepared, version = "" }) { const { rn, loader, rule, mem, ram, disk } = prepared; const lib = { MODELS: rn.MODELS, FILES: rn.FILES, NEED_GB: rn.NEED_GB, roomBytes: rn.roomBytes, roomFit: rn.roomFit, shortNote: rn.shortNote, shortBy: rn.shortBy, gbUp: rn.gbUp, pledgeGB: rn.pledgeGB, nodeCtxFor: rn.nodeCtxFor, dealRoom: rn.dealRoom, - pickCtx: rn.pickCtx, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor, ngramFor: rn.ngramFor, ngramOf: rn.ngramOf }; + pickCtx: rn.pickCtx, roomCtxPlan: rn.roomCtxPlan, maxSeqFor: rn.maxSeqFor, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor, ngramFor: rn.ngramFor, ngramOf: rn.ngramOf }; const dir = opts.modelDir; const keys = hostable(rn.MODELS); const pulled = new Set(keys.filter((k) => modelState(dir, k, rn.MODELS, rn.FILES, rn.LOCAL).pulled)); diff --git a/cli/lib/hostui.js b/cli/lib/hostui.js index e6b62033..5d1c24a0 100644 --- a/cli/lib/hostui.js +++ b/cli/lib/hostui.js @@ -31,8 +31,10 @@ export function deviceKind(meta = {}, self = false) { // (roomBytes), else room/models.js NEED_GB for a model without a SHAPE // rounded up to a tenth of a GiB, as room/models.js NEED_GB (the room page's number for the same model) const needAt = (lib, key, ctx) => { const rb = lib.roomBytes(key, ctx, "f16"); return rb ? Math.ceil((rb.L * rb.layerBytes + rb.hostBytes) / GiB * 10 - 1e-9) / 10 : null; }; +// (without --ctx: at the model's default, CTX def, as the room page's NEED_GB: a model with a ladder opens longer only +// when the room holds that at full speed, room/ctxpick.js roomCtxPlan) export function modelNeedGB(lib, key, ctxAsk = 0) { - return needAt(lib, key, lib.nodeCtxFor(key, ctxAsk)) ?? lib.NEED_GB[key] ?? null; + return needAt(lib, key, lib.maxSeqFor ? lib.maxSeqFor(key, ctxAsk) : lib.nodeCtxFor(key, ctxAsk)) ?? lib.NEED_GB[key] ?? null; } // The context a node falls back to for `key` when the room is short for its default (room/models.js // pickCtx: the 1.7B opens at 8K when 16K doesn't fit), and what the model needs there; null when it @@ -41,7 +43,11 @@ export function modelFallback(lib, key, ctxAsk = 0) { if (!lib.pickCtx) return null; const want = lib.nodeCtxFor(key, ctxAsk); const p = lib.pickCtx(key, { want, ask: ctxAsk, fitsAt: () => false }); - return p.ctx < want ? { ctx: p.ctx, needGB: needAt(lib, key, p.ctx) } : null; + if (!(p.ctx < want)) return null; + // (a fallback that saves under 1 GB is not worth a word: the 27B's and the MoE's 8K, ~0.5 GB under their default; the + // room still opens there when that is all it holds) + const needGB = needAt(lib, key, p.ctx), def = modelNeedGB(lib, key, ctxAsk); + return needGB != null && def != null && def - needGB < 1 ? null : { ctx: p.ctx, needGB }; } const kOf = (n) => `${Math.round(n / 1024)}K`; // "6 GB" or "6 GB (4 GB at 8K)": a model's need in words, whole GB @@ -74,11 +80,15 @@ export function roomFitNow(lib, { model, devices, ctxAsk = 0, spareGB = [] }) { } const off = (rb) => (lib.offloadFor ? lib.offloadFor(devices.map((d) => d.meta), rb.experts || rb.expertBytes || 0, model) : null); // (rb.layerCost: Qwen3.8-Flash-Next's per-layer bytes, room/models.js flashLayerCost, which roomFit takes in place of one size) - const fitAt = (c) => { const rb = lib.roomBytes(model, c, "f16"); return lib.roomFit(rb.L, pl.map((g) => g * GiB), rb.layerCost || rb.layerBytes, rb.hostBytes, off(rb)); }; - const pick = lib.pickCtx ? lib.pickCtx(model, { want, ask: ctxAsk, fitsAt: (c) => fitAt(c).fits }) : { ctx: want, want, fellBack: false }; + // (lib.roomCtxPlan: the room node's own plan, room/ctxpick.js, so the screen opens the room where Start will: the top of + // the model's ladder down to what fits, past its default only at full speed) + const plan = lib.roomCtxPlan ? lib.roomCtxPlan(model, { metas: devices.map((d) => d.meta), pledges: pl.map((g) => g * GiB), ask: ctxAsk }) : null; + const fitAt = plan ? (c) => plan.fitAt(c).fit + : (c) => { const rb = lib.roomBytes(model, c, "f16"); return lib.roomFit(rb.L, pl.map((g) => g * GiB), rb.layerCost || rb.layerBytes, rb.hostBytes, off(rb)); }; + const pick = plan || (lib.pickCtx ? lib.pickCtx(model, { want, ask: ctxAsk, fitsAt: (c) => fitAt(c).fits }) : { ctx: want, want, fellBack: false }); const fb = modelFallback(lib, model, ctxAsk); - const base = { needGB, minGB: fb?.needGB ?? null, haveGB, ctx: pick.ctx, want, fellBack: !!pick.fellBack, - ctxNote: pick.fellBack && lib.ctxShortNote ? lib.ctxShortNote(label, pick.ctx, want) : "" }; + const base = { needGB, minGB: fb?.needGB ?? null, haveGB, ctx: pick.ctx, want: pick.want ?? want, fellBack: !!pick.fellBack, + ctxNote: pick.fellBack && lib.ctxShortNote ? lib.ctxShortNote(label, pick.ctx, pick.want ?? want) : "" }; const fit = fitAt(pick.ctx); if (fit.fits) return { fits: true, ...base, shortGB: 0, note: "", offload: !!fit.offload, disk: !!fit.disk }; const shortGB = lib.gbUp(lib.shortBy ? lib.shortBy(fit, spareGB) : fit.short); diff --git a/cli/lib/lend.js b/cli/lib/lend.js index 72fe1781..cde092bd 100644 --- a/cli/lib/lend.js +++ b/cli/lib/lend.js @@ -126,9 +126,9 @@ export const HELP_HOST = `Usage More options --code the room code to use (default: a random one of six characters) - --ctx the context to ask for, in tokens (default: the longest the model takes in a - room: 16k on the 1.7B, 8k when the room is short of memory for 16k; 64k on - the 27B, 128k on the MoE) + --ctx the context to ask for, in tokens (default: the longest the room holds at full + speed: up to 64k on the 27B and 128k on the MoE, 16k on the 1.7B; shorter + when the room is short of memory) --devices the same as --wait (its older name) --signal PeerJS signaling server(s), as pooled join --no-check skip the test allocation that confirms the memory is there diff --git a/cli/lib/lendrun.js b/cli/lib/lendrun.js index 108801b9..35b95cfa 100644 --- a/cli/lib/lendrun.js +++ b/cli/lib/lendrun.js @@ -472,7 +472,7 @@ async function runHost(opts, out, prepared = null) { // --here: lend what the model needs on this computer alone and start; --pool: spread over the room if (opts.mode === "here") { const { hereGB, hereWhy, modelNeedGB, modelFallback } = await import("./hostui.js"); - const lib = { MODELS: rn.MODELS, NEED_GB: rn.NEED_GB, roomBytes: rn.roomBytes, roomFit: rn.roomFit, shortNote: rn.shortNote, shortBy: rn.shortBy, gbUp: rn.gbUp, pledgeGB: rn.pledgeGB, nodeCtxFor: rn.nodeCtxFor, pickCtx: rn.pickCtx, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor }; + const lib = { MODELS: rn.MODELS, NEED_GB: rn.NEED_GB, roomBytes: rn.roomBytes, roomFit: rn.roomFit, shortNote: rn.shortNote, shortBy: rn.shortBy, gbUp: rn.gbUp, pledgeGB: rn.pledgeGB, nodeCtxFor: rn.nodeCtxFor, pickCtx: rn.pickCtx, roomCtxPlan: rn.roomCtxPlan, maxSeqFor: rn.maxSeqFor, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor }; const max = Math.max(rule.gb, memoryRule(mem, { max: true }, { capGB: rn.maxPledgeGB(opts.model) }).gb || rule.gb); // a MoE model this GPU can't hold whole: with --ram (a discrete GPU's default), its experts in RAM, lending the rule's GB // --disk: the model's files will be here (pulled below), unless --no-pull streams a model that is not @@ -598,7 +598,7 @@ async function runHost(opts, out, prepared = null) { // model, by the room page's math (room/plan.js roomFit); a terminal otherwise waits for Enter const auto = !tty || opts.start || opts.devices > 0; const lib = { MODELS: rn.MODELS, NEED_GB: rn.NEED_GB, roomBytes: rn.roomBytes, roomFit: rn.roomFit, shortNote: rn.shortNote, shortBy: rn.shortBy, - gbUp: rn.gbUp, pledgeGB: rn.pledgeGB, nodeCtxFor: rn.nodeCtxFor, pickCtx: rn.pickCtx, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor }; + gbUp: rn.gbUp, pledgeGB: rn.pledgeGB, nodeCtxFor: rn.nodeCtxFor, pickCtx: rn.pickCtx, roomCtxPlan: rn.roomCtxPlan, maxSeqFor: rn.maxSeqFor, ctxShortNote: rn.ctxShortNote, offloadFor: rn.offloadFor }; let lastWhy = ""; const maybeDeal = () => { if (!auto || starting || leaving || node.ai.engine || node.ai.starting) return; diff --git a/cli/test/hostui_test.mjs b/cli/test/hostui_test.mjs index be70891e..3f224c38 100644 --- a/cli/test/hostui_test.mjs +++ b/cli/test/hostui_test.mjs @@ -2,7 +2,8 @@ // with the room page's math, Start gating, the keys, and flags that skip every question. import { test } from "node:test"; import assert from "node:assert/strict"; -import { MODELS, FILES, NEED_GB, roomBytes, pickCtx, ctxShortNote } from "../../room/models.js"; +import { MODELS, FILES, NEED_GB, roomBytes, pickCtx, ctxShortNote, maxSeqFor } from "../../room/models.js"; +import { roomCtxPlan } from "../../room/ctxpick.js"; import { roomFit, shortNote, shortBy, gbUp } from "../../room/plan.js"; import { pledgeGB } from "../../room/pledge.js"; import { nodeCtxFor } from "../../packages/room-node/roomnode.js"; @@ -11,7 +12,8 @@ import { width } from "../lib/style.js"; import { keysOf } from "../lib/tui.js"; import { parseLendArgs } from "../lib/lend.js"; -const lib = { MODELS, FILES, NEED_GB, roomBytes, roomFit, shortNote, shortBy, gbUp, pledgeGB, nodeCtxFor, pickCtx, ctxShortNote }; +// (as cli/lib/hostrun.js builds it from the room node: roomCtxPlan is the room's own context plan, room/ctxpick.js) +const lib = { MODELS, FILES, NEED_GB, roomBytes, roomFit, shortNote, shortBy, gbUp, pledgeGB, nodeCtxFor, pickCtx, roomCtxPlan, maxSeqFor, ctxShortNote }; const KEYS = ["qwen3-0.6b", "qwen3-1.7b", "qwen3-4b", "qwen3.8-27b", "qwen3.6-35b-moe"]; const dev = (name, gb, extra = {}) => ({ name, meta: { webgpu: true, contribGB: gb, ...extra } }); @@ -41,10 +43,11 @@ test("a short room: the same sentence and number as the room page; pledges that const moe = "qwen3.6-35b-moe"; const one = roomFitNow(lib, { model: moe, devices: [dev("spark", 12)], spareGB: [0] }); assert.equal(one.fits, false); - assert.ok(one.shortGB > 9 && one.shortGB < 15, `short ${one.shortGB}`); + assert.ok(one.shortGB > 7 && one.shortGB < 15, `short ${one.shortGB}`); assert.match(one.note, /^This room is [\d.]+ GB short for Qwen3.6 35B MoE: add a device or raise a pledge/); - // the room page's own computation, byte for byte - const rb = roomBytes(moe, nodeCtxFor(moe), "f16"); + // the room page's own computation, byte for byte: short everywhere on the MoE's ladder, so short for its last step (8K) + assert.equal(one.ctx, 8192); + const rb = roomBytes(moe, 8192, "f16"); assert.equal(one.shortGB, gbUp(roomFit(rb.L, [12 * 2 ** 30], rb.layerBytes, rb.hostBytes).short)); const two = roomFitNow(lib, { model: moe, devices: [dev("spark", 12), dev("mac", 16, { native: "node-dawn" })] }); assert.equal(two.fits, true); assert.equal(two.haveGB, 28); assert.equal(two.note, ""); @@ -177,7 +180,7 @@ test("the screen: fits 58, 80 and 110 columns in every style, the room's devices assert.match(text, /Qwen3\.6 35B MoE · Q4 · waiting for devices/); assert.match(text, /DEVICE\s+GPU\s+LENDS/); assert.match(text, /node-abc\s+4 GB\s+joined/); assert.match(text, /spark\s+12 GB\s+this computer/); assert.match(text, /otter\s+Mac, 8 GB\s+wants to join a allow d deny/); - assert.match(text, /memory ━+─* 17 GB lent · 23 GB needed/); + assert.match(text, /memory ━+─* 17 GB lent · 21 GB needed/); assert.match(text, /invite pooled join 4TK-G9P on the other computer/); assert.match(text, /Needs \d+ GB more: one more device, or press l to lend more\./); assert.match(text, /enter start · i copy invite · m model · l lend · s split · q quit/); @@ -246,7 +249,7 @@ test("the host screen: the header once (pill, link, status), the picker as one p const text = L.join("\n"); assert.match(text, /Which model\?\n\n Qwen3 0\.6B/, "one list, smallest first, no groups"); assert.doesNotMatch(text, /Fits on this computer|Needs another device/); - assert.match(text, /\n Qwen3\.8 27B\s+needs 20 GB\s+15\.0 GB download/); + assert.match(text, /\n Qwen3\.8 27B\s+needs 17 GB\s+15\.0 GB download/); assert.match(text, /\n Qwen3 1\.7B\s+needs 6 GB \(4 GB at 8K\) 1\.7 GB download/, "the 1.7B: 16K, and its 8K fallback"); assert.match(text, /› Qwen3 4B\s+needs 5 GB/, "the recommended model is preselected, not labelled"); assert.doesNotMatch(text, /recommended|"needs"/); @@ -366,9 +369,9 @@ test("how to run it: Pool with devices first and preselected; Run it here lends // a model this computer can't hold: Run it here is off, with why s.sel = rows.indexOf(moe); r = reduce(s, "enter"); r = reduce(r.state, "down"); - assert.match(render(r.state, { lib }).join("\n"), /Run it here +needs 23 GB, this computer has 14 GB to lend/); + assert.match(render(r.state, { lib }).join("\n"), /Run it here +needs 21 GB, this computer has 14 GB to lend/); r = reduce(r.state, "enter"); - assert.equal(r.state.step, "how"); assert.match(r.state.notice, /needs 23 GB, this computer has 14 GB to lend/); + assert.equal(r.state.step, "how"); assert.match(r.state.notice, /needs 21 GB, this computer has 14 GB to lend/); r = reduce(r.state, "up"); r = reduce(r.state, "enter"); assert.equal(r.state.step, "pledge"); assert.equal(r.state.splitMode, "memory"); }); @@ -473,7 +476,9 @@ test("expert offload: a 12 GB GPU with RAM to spare runs the 35B --here, its exp const olib = { ...lib, offloadFor }; // the RTX 5070 PC: lends 10 GB by the memory rule (11 at most), 48 GB of its 64 GB of RAM (total less 16) assert.equal(hereGB(olib, "qwen3.6-35b-moe", { maxGB: 11 }), null, "no RAM: as before"); - assert.equal(hereGB(lib, "qwen3.6-35b-moe", { maxGB: 11, ramGB: 48, offGB: 10 }), null, "a lib without offloadFor: as before"); + // (an older room node's lib: no offloadFor and no roomCtxPlan, which brings the room's own offload rules) + const old = { ...lib, roomCtxPlan: undefined, maxSeqFor: undefined }; + assert.equal(hereGB(old, "qwen3.6-35b-moe", { maxGB: 11, ramGB: 48, offGB: 10 }), null, "a lib without offloadFor: as before"); assert.equal(hereGB(olib, "qwen3.6-35b-moe", { maxGB: 11, ramGB: 48, offGB: 10 }), 10, "the memory rule's 10 GB, the experts it can't hold in RAM"); assert.equal(hereGB(olib, "qwen3.6-35b-moe", { maxGB: 11, ramGB: 4, offGB: 10 }), null, "4 GB of RAM can't take the experts"); // a model it holds whole is still dealt whole, at the least that holds it @@ -503,3 +508,18 @@ test("--here --disk: a MoE model the GPU and --ram can't hold runs here with som assert.deepEqual(selfMeta(10, 8, M), { contribGB: 10, webgpu: true, offload: true, ramGB: 8, disk: true, local: [M] }); assert.deepEqual(selfMeta(10, 0, M), { contribGB: 10, webgpu: true }, "no RAM: no disk tier"); }); + +test("the 27B and the MoE: pooled host's screen, the room node and the room page open a room at the same context", async () => { + const { RoomNode } = await import("../../packages/room-node/roomnode.js"); + const { dealPledgeBytes } = await import("../../room/pledge.js"); + const rooms = [[dev("mac", 16), dev("pc", 7, { native: "node-dawn" }), dev("iphone", 1, { ua: "iPhone" })], [dev("mac", 17), dev("pc", 6), dev("iphone", 1, { ua: "iPhone" })], + [dev("spark", 40, { native: "node-dawn" })], [dev("pc", 12, { native: "node-dawn", offload: true, ramGB: 48 })], [dev("mac", 16)], [dev("a", 11), dev("b", 11)]]; + for (const model of ["qwen3.8-27b", "qwen3.6-35b-moe"]) for (const devices of rooms) { + const at = `${model}: ${devices.map((d) => `${d.name} ${d.meta.contribGB}`).join(" + ")}`; + const screen = roomFitNow(lib, { model, devices }); + const page = roomCtxPlan(model, { metas: devices.map((d) => d.meta), pledges: devices.map((d) => dealPledgeBytes(d.meta, Infinity, model)) }); + const node = RoomNode.ctxPick(model, { want: nodeCtxFor(model), self: { name: devices[0].name, meta: devices[0].meta }, peers: devices.slice(1).map((d, i) => ({ id: String(i), name: d.name, meta: d.meta })) }); + assert.deepEqual([screen.ctx, screen.fits, screen.fellBack], [page.ctx, page.fits, page.fellBack], `${at}: screen`); + assert.deepEqual([node.ctx, node.fits, node.fellBack], [page.ctx, page.fits, page.fellBack], `${at}: node`); + } +}); diff --git a/docs-site/src/content/docs/code/agent.mdx b/docs-site/src/content/docs/code/agent.mdx index fa5adbb4..e44b84a7 100644 --- a/docs-site/src/content/docs/code/agent.mdx +++ b/docs-site/src/content/docs/code/agent.mdx @@ -145,7 +145,7 @@ The agent's conversation must fit the model's context, less room for the next an | Qwen3.8 27B | 16,384 | about 12,220 | | Qwen3.6 35B MoE | 32,768 | about 24,510 | -The meter next to **New task** shows how much of the context the conversation uses. A bigger `?ctx=` gives the agent more room; see [Pick a model](/docs/models). +The meter next to **New task** shows how much of the context the conversation uses. A longer context gives the agent more room: the 27B and the MoE open at the longest the room holds, and the picker's **Context** row or `?ctx=` sets another; see [Longer context](/docs/models#longer-context). When the conversation grows past its budget, the agent shortens it to 60% of the budget, in this order: diff --git a/docs-site/src/content/docs/contributing/models.mdx b/docs-site/src/content/docs/contributing/models.mdx index 90ae5bec..78a90f38 100644 --- a/docs-site/src/content/docs/contributing/models.mdx +++ b/docs-site/src/content/docs/contributing/models.mdx @@ -82,14 +82,15 @@ A new architecture or quantization format needs a design note first. See [Contri ## Context length -The context a room opens with comes from `CTX` in `room/models.js`, and `?ctx=N` can raise it up to `max`: +The context a room opens with comes from `CTX` in `room/models.js`, and `?ctx=N` can ask for any length up to `max`: -| Model | Default | Max | -|---|---|---| -| Qwen 3.8 27B | 16384 | 32768 | -| Qwen 3.6 35B MoE | 32768 | 65536 | -| Qwen3 1.7B | 16384 (`fallback`: 8192) | 16384 | +| Model | Default (`def`) | Ladder | Picker (`options`) | Max | +|---|---|---|---|---| +| Qwen 3.8 27B | 16384 | 65536, 32768, 16384, 8192 | 8K, 16K, 32K, 64K | 65536 | +| Qwen 3.6 35B MoE | 32768 | 131072, 65536, 32768, 16384, 8192 | 8K, 32K, 64K, 128K | 131072 | +| Qwen3.8-Flash-Next | 65536 | 65536 ... 2048 (`FLASH_CTX_PROVEN`) | 8K, 32K, 64K | 65536 | +| Qwen3 1.7B | 16384 (`fallback`: 8192) | none | none | 16384 | -A `fallback` is the context a room opens with when its pledges can't hold the model at the default but can there (`pickCtx`): the room page, `pooled host` and the OpenClaw plugin all pick it the same way, and say so. `NEED_MIN_GB` is the need at the fallback. An explicit `?ctx=` or `--ctx` is never lowered. +A `ladder` is the lengths a room may open the model at, longest first; put a length on it only once it is checked correct on that model (`docs/long-context-and-sessions.md`). `room/ctxpick.js` `roomCtxPlan` picks one: the longest step the room's pledges hold, but past `def` only at full speed (the deal by memory keeps every expert on a GPU and uses no more devices than at `def`). A `fallback` is the context a room opens with when its pledges can't hold the model at the default but can there. The room page (picker and Start), the room node (`pooled host`, `pooled serve`'s room, the OpenClaw plugin) and `pooled host`'s screen all call `roomCtxPlan`, and say so when they fall back. `NEED_GB` is the need at `def`, `NEED_MIN_GB` at the fallback or the ladder's last step. An explicit `?ctx=` or `--ctx` is tried alone and never lowered. Without a `CTX` entry a model gets 8192 tokens on the hybrid engine and 2048 on the dense engine. The hybrid models keep a KV cache only on their full-attention layers, so long context is cheap in memory; the practical limit is prefill speed. See [Long context and sessions](/docs/internals/sessions). diff --git a/docs-site/src/content/docs/internals/sessions.md b/docs-site/src/content/docs/internals/sessions.md index 39b9b284..b2ee3c25 100644 --- a/docs-site/src/content/docs/internals/sessions.md +++ b/docs-site/src/content/docs/internals/sessions.md @@ -12,7 +12,7 @@ A coding agent resends almost its whole conversation on every turn: about 96% of ## Context per model -Each model's default and maximum context are on [Models reference](/docs/reference/models). `?ctx=` on the model host's page asks for another size, rounded to a multiple of 256, at least 2,048 and at most the maximum. The KV cache is split with the layers, so each device holds only its own attention layers' share, and the practical limit is prefill time, not memory. +Each model's default and maximum context are on [Models reference](/docs/reference/models). The 27B and the MoE open at the longest length of their ladder the room holds at full speed ([Longer context](/docs/models#longer-context)). `?ctx=` on the model host's page asks for another size, rounded to a multiple of 256, at least 2,048 and at most the maximum. The KV cache is split with the layers, so each device holds only its own attention layers' share, and the practical limit is prefill time, not memory. ## The KV cache diff --git a/docs-site/src/content/docs/kits/coding-tools.mdx b/docs-site/src/content/docs/kits/coding-tools.mdx index 7b95b2ad..5fcb9947 100644 --- a/docs-site/src/content/docs/kits/coding-tools.mdx +++ b/docs-site/src/content/docs/kits/coding-tools.mdx @@ -47,14 +47,12 @@ Everything the tool sends, file contents included, goes to the room's host and t -1. **Start the room with a long context.** On the device that will host, open the room with `?ctx=` before you press **Start the model**: +1. **Start the room.** On the device that will host, open [pooled.run/room](https://pooled.run/room), pick **Qwen3.6 35B MoE** and press **Start the model**. It opens at the longest context the room holds, 64K or 128K in most rooms that fit it; the **Context** row under the models shows which. To pin an exact length, such as the 64K in the example below, open the room with `?ctx=` before you press Start: ```text https://pooled.run/room?ctx=65536 ``` - Pick **Qwen3.6 35B MoE** and press **Start the model**. (For the 27B, use `?ctx=32768`.) - 2. **Copy the command.** In the room, press **Serve API** in the header. The page shows the command for this room with a **Copy** button. On the device that opened the room, it carries the invite link: ```bash diff --git a/docs-site/src/content/docs/kits/terminal.mdx b/docs-site/src/content/docs/kits/terminal.mdx index 0dbd98fd..8fd2d7c0 100644 --- a/docs-site/src/content/docs/kits/terminal.mdx +++ b/docs-site/src/content/docs/kits/terminal.mdx @@ -128,7 +128,7 @@ Sources (for the fact-checker): - After step 7: `● online` at the top and `room online: spark-host 0-21 · spark-join 22-39` in the log, about a minute after Enter when the model is on disk (63 seconds in our test: each computer loads its part of the 19.4 GB file). - After step 8: an answer, then a line like `74 tokens · 46 tok/s · first token 0.4s`. -What a model **needs** grows with the room's context. `pooled host` opens each model at its longest room context: 128k on the MoE, which needs 23 GB, and 16k on the 1.7B, which an agent such as OpenClaw needs. The 1.7B needs 5.5 GB at 16k, shown as `needs 6 GB (4 GB at 8K)`. When the room has less, `pooled host` opens it with an 8k context instead and says so: `Qwen3 1.7B · 8K context: the room's memory is short for 16K`. `--ctx 16384` or `--ctx 8192` takes that context and no other. +What a model **needs** grows with the room's context. `pooled host` opens the 27B and the MoE at the longest context the room holds without slowing down: up to 64k on the 27B and 128k on the MoE (the MoE needs 21 GB at its 32k default, 23 GB at 128k), down to 8k when the room is short. It opens the 1.7B at 16k, which an agent such as OpenClaw needs. The 1.7B needs 5.5 GB at 16k, shown as `needs 6 GB (4 GB at 8K)`. When the room has less, `pooled host` opens it with an 8k context instead and says so: `Qwen3 1.7B · 8K context: the room's memory is short for 16K`. `--ctx 16384` or `--ctx 8192` takes that context and no other. ## Chat from anywhere diff --git a/docs-site/src/content/docs/models.mdx b/docs-site/src/content/docs/models.mdx index 24406428..215d4afe 100644 --- a/docs-site/src/content/docs/models.mdx +++ b/docs-site/src/content/docs/models.mdx @@ -9,6 +9,9 @@ sidebar: {/* Sources (for the fact-checker): - room/models.js: PICKER, NEED_GB (5.6 / 17.0 / 22.5), NEED_MIN_GB (1.7B 4.0 at its 8K fallback), CTX.fallback + pickCtx() (16K, else 8K when the pledges are short for 16K), FILE_GB (1.83 / 16.06 / 20.84), CTX (1.7B 16384/16384, fallback 8192, 27B 16384/32768, MoE 32768/65536), maxSeqFor() (?ctx= rounded to 256, at least 2048) +- context ladder (feat/ctx-default-ladder): CTX ladder / options for the 27B (64K/32K/16K/8K; picker 8K/16K/32K/64K) and the MoE + (128K/64K/32K/16K/8K; picker 8K/32K/64K/128K), room/ctxpick.js roomCtxPlan (past def only with no expert offloaded and on no + more devices than at def), NEED_MIN_GB at 8K (27B 16.5, MoE 22.0) - 0.4.0 (origin/main ed31899): CTX 27B max 65536, MoE max 131072; the 122B (#316): NEED_GB 72 (the CLI picker's "needs 73 GB" with its context), FILE_GB 72.52, in hostable() for pooled host / pull, ?dev=1 only on the room page; expert offload (#319, #320, #323): 35B on the RTX 5070 alone about 38-45 tok/s (37.5-44.9 measured on the published 0.4.0, ~40 on long generations), 17.5 GiB parked; @@ -28,14 +31,14 @@ Pick by how much memory the room has. The header shows it as `N GB pooled`, and | Model | The room needs | Download | Context | |---|---|---|---| | Qwen3 1.7B | 5.6 GB (4 GB at 8K) | 1.83 GB | 16K (8K when the room is short) | -| Qwen3.8 27B | 17 GB | 16.06 GB | 16K (up to 64K) | -| Qwen3.6 35B MoE | 22.5 GB | 20.84 GB | 32K (up to 128K) | +| Qwen3.8 27B | 17 GB (16.5 GB at 8K) | 16.06 GB | up to 64K: the longest the room holds (16K at 17 GB, 64K at about 20 GB) | +| Qwen3.6 35B MoE | 22.5 GB (22 GB at 8K) | 20.84 GB | up to 128K: the longest the room holds (32K at 22.5 GB, 128K at about 23.5 GB) | | Qwen3.5 122B MoE (preview: `?dev=1`, or the terminal) | 73 GB | 72.52 GB | 32K (up to 128K) | | Qwen3.8-Flash-Next (**experimental**; `POOLED_DEV=1` for `pooled host` / `list`) | 72 GB, or about 8 GB on one tab with its experts on disk | 100.6 GB (71.8 GB of layers, 28.8 GB n-gram table) | 64K (the picker offers 8K, 32K and 64K; 128K in progress) | The download is split across the devices that run the model, so each one fetches only its part. -The 1.7B opens with a 16K context when the room has 5.6 GB for it. With less, down to 4 GB, it opens with 8K instead, and the room says so: `Qwen3 1.7B · 8K context: the room's memory is short for 16K`. The picker row then reads `4 GB · 8K fits`. Add a device or raise a pledge for 16K. A room opened with `?ctx=` keeps the context it asked for. +The 27B and the MoE open at the longest context the room holds without slowing down (see [Longer context](#longer-context)). The 1.7B opens with a 16K context when the room has 5.6 GB for it. With less, down to 4 GB, it opens with 8K instead, and the room says so: `Qwen3 1.7B · 8K context: the room's memory is short for 16K`. The picker row then reads `4 GB · 8K fits`. Add a device or raise a pledge for 16K. A room opened with `?ctx=` keeps the context it asked for. ## Which one to pick @@ -75,13 +78,25 @@ Only give memory the device really has free. On a Mac, the GPU shares memory wit ## Longer context -Context is the prompt and the answer together. To get more, add `?ctx=` to the room's address before you press Start, on the device that will host. For example, 64K on the MoE: +Context is the prompt and the answer together. The 27B, the MoE and Qwen3.8 Flash open at the longest context the room's memory holds, so you don't need to set anything: + +| Model | Lengths | Opens at | +|---|---|---| +| Qwen3.8 27B | 8K, 16K, 32K, 64K | the longest that fits, at least 16K when the room holds that | +| Qwen3.6 35B MoE | 8K, 16K, 32K, 64K, 128K | the longest that fits, at least 32K when the room holds that | +| Qwen3.8 Flash | 2K to 64K | the longest that fits | + +A longer context only takes memory: a reply comes just as fast at 128K as at 32K until the conversation actually gets that long. So the room goes past the 27B's 16K or the MoE's 32K only when that costs no speed: every device keeps its layers' experts on its GPU, and no extra device (such as a phone) has to hold layers. Otherwise it stays at 16K or 32K. A room short of memory even for those opens at a shorter length and says so, for example `Qwen3.6 35B MoE · 8K context: the room's memory is short for 32K`. + +The picker shows a **Context** row under the models with each length, the memory it needs in this room, and which ones fit. Pick one to open the room at it (the room remembers your choice). While the model runs, **Room settings** can change it; the model then loads again on every device. Reading a long prompt from scratch still takes a while: about 2 minutes for 32K tokens and 15 minutes for 128K on the MoE on one GB10. + +To ask for an exact length instead, add `?ctx=` to the room's address before you press Start, on the device that will host. For example, 64K on the MoE: ```text https://pooled.run/room?ctx=65536 ``` -See [URL options](/docs/reference/url-options) for this and `?kv=q8`, which saves memory on the two big models. +The room then opens at that length or not at all. See [URL options](/docs/reference/url-options) for this and `?kv=q8`, which saves memory on the two big models. ## Details diff --git a/docs-site/src/content/docs/reference/models.mdx b/docs-site/src/content/docs/reference/models.mdx index e87f74b5..4be4ae62 100644 --- a/docs-site/src/content/docs/reference/models.mdx +++ b/docs-site/src/content/docs/reference/models.mdx @@ -26,10 +26,10 @@ Pooled ships no weights. Each device downloads only its share of the layers from | Model | Quantization | File | Memory the room needs | Context (default / max) | |---|---|---|---|---| | Qwen3 1.7B | Q8_0 | 1.83 GB | 4.0 GB | 8192 / 16384 | -| Qwen 3.8 27B | Q4_0 | 16.06 GB | 17.0 GB | 16384 / 65536 | -| Qwen 3.6 35B MoE (35B-A3B) | Q4_0 | 20.84 GB | 22.5 GB | 32768 / 131072 | +| Qwen 3.8 27B | Q4_0 | 16.06 GB | 17.0 GB | 16384 / 65536 (opens up to 65536) | +| Qwen 3.6 35B MoE (35B-A3B) | Q4_0 | 20.84 GB | 22.5 GB | 32768 / 131072 (opens up to 131072) | -"Memory the room needs" is the total across all devices. Raise the context with [`?ctx=`](/docs/reference/url-options#context-per-model). +"Memory the room needs" is the total across all devices, at the default context. The 27B and the MoE open longer when the room holds that at full speed: see [Longer context](/docs/models#longer-context). Ask for an exact length with [`?ctx=`](/docs/reference/url-options#context-per-model). ### What each one is diff --git a/docs-site/src/content/docs/reference/url-options.mdx b/docs-site/src/content/docs/reference/url-options.mdx index f1173fa7..7526343f 100644 --- a/docs-site/src/content/docs/reference/url-options.mdx +++ b/docs-site/src/content/docs/reference/url-options.mdx @@ -29,7 +29,7 @@ Supported and stable. | Option | Default | Where | What it does | |---|---|---|---| -| `?ctx=N` | per model | Host | Context window in tokens. Rounded to 256, at least 2048, at most the model's cap (below). | +| `?ctx=N` | per model | Host | Context window in tokens: exactly this length, instead of the longest the room holds. Rounded to 256, at least 2048, at most the model's cap (below). | | `?kv=q8` | off (f16) | Host | int8 KV cache, about 56% of the memory. 27B and MoE only. Prefill of long prompts gets much slower. | | `?split=speed` or `memory` | `speed` | Host | Layer split, same as the room setting. See [How layers are split](/docs/rooms/split). | | `?phonelayers=1` | off | Host | Gives phones layers even when the computers can hold the model. | @@ -47,16 +47,18 @@ Supported and stable. ### Context per model +Without `?ctx=` the 27B and the MoE open at the longest length of their ladder the room holds at full speed (every expert on a GPU, no extra device holding layers), at least their old default when the room holds that; the picker's **Context** row picks another. See [Longer context](/docs/models#longer-context). + | Model | Default | Largest `?ctx=` | |---|---|---| | Qwen3 1.7B | 16384 (8192 when the room is short of memory) | 16384 | -| Qwen 3.8 27B | 16384 | 65536 | -| Qwen 3.6 35B MoE | 32768 | 131072 | +| Qwen 3.8 27B | the longest of 65536, 32768, 16384 the room holds at full speed (8192 when the room is short of memory for 16384) | 65536 | +| Qwen 3.6 35B MoE | the longest of 131072, 65536, 32768 the room holds at full speed (16384 or 8192 when the room is short of memory for 32768) | 131072 | | Qwen3.5 122B MoE (`?dev=1`) | 32768 | 131072 | | Qwen3.8-Flash-Next (experimental) | 65536 (a room short of memory opens at the longest proven length it holds: 32768, 16384, 8192 or 2048) | 65536 (128K is in progress) | | Other models (tests and `?dev=1`) | 2048 (8192 on the hybrid engine) | same | -Long context costs little memory on the 27B and MoE; what limits it is how long the prompt takes to read. See [Long context and sessions](/docs/internals/sessions). +Long context costs little memory on the 27B and MoE (64K adds about 3 GB to the 27B's 16K, 128K about 2 GB to the MoE's 32K); what limits it is how long the prompt takes to read. See [Long context and sessions](/docs/internals/sessions). ## Room links diff --git a/docs-site/src/content/docs/serve/index.mdx b/docs-site/src/content/docs/serve/index.mdx index a1ef8660..69eebf6a 100644 --- a/docs-site/src/content/docs/serve/index.mdx +++ b/docs-site/src/content/docs/serve/index.mdx @@ -94,7 +94,7 @@ print(m.content[0].text) ## Pick a model for agents -Coding agents send long prompts (15,000 to 20,000 tokens for Codex and Claude Code) and need a model that calls tools reliably. Use the **Qwen3.6 35B MoE** (32K context, up to 64K) or the **Qwen3.8 27B** at 32K. The Qwen3 1.7B is too small for them. See [Limits and context](/docs/serve/limits). +Coding agents send long prompts (15,000 to 20,000 tokens for Codex and Claude Code) and need a model that calls tools reliably. Use the **Qwen3.6 35B MoE** or the **Qwen3.8 27B**: rooms open them at the longest context their memory holds (up to 128K and 64K). The Qwen3 1.7B is too small for them. See [Limits and context](/docs/serve/limits). :::caution[Your prompts go to the room] Everything a tool sends goes to the room's host and the devices that run the model. Coding tools send file contents and sometimes secrets. Only use a room you trust. See [Who sees your prompts](/docs/serve/options#who-sees-your-prompts). diff --git a/docs-site/src/content/docs/serve/limits.md b/docs-site/src/content/docs/serve/limits.md index 96172878..a1b632a4 100644 --- a/docs-site/src/content/docs/serve/limits.md +++ b/docs-site/src/content/docs/serve/limits.md @@ -18,10 +18,10 @@ The room's context is the most tokens one request can hold: prompt plus answer. | Model | Default | Most with `?ctx=` | |---|---|---| | Qwen3 1.7B | 16384 (8192 when the room is short of memory for 16K) | 16384 | -| Qwen3.8 27B | 16384 | 32768 | -| Qwen3.6 35B MoE | 32768 | 65536 | +| Qwen3.8 27B | up to 65536: the longest the room holds at full speed, at least 16384 when it holds that | 65536 | +| Qwen3.6 35B MoE | up to 131072: the longest the room holds at full speed, at least 32768 when it holds that | 131072 | -The host sets it by opening the room page with `?ctx=` before starting the model, for example `pooled.run/room?ctx=65536`. The value is rounded to a multiple of 256, at least 2048. +Most rooms that hold the 27B or the MoE open them at 64K or more on their own (see [Longer context](/docs/models#longer-context)); the picker's **Context** row picks another length. To ask for an exact one, the host opens the room page with `?ctx=` before starting the model, for example `pooled.run/room?ctx=65536`. The value is rounded to a multiple of 256, at least 2048. The bridge reports it in the banner, as `ctx` in `GET /health` and as `max_model_len` in `GET /v1/models`. Give it to coding agents so they compact in time: see [context settings](/docs/serve/recipes#context-settings-for-coding-agents). diff --git a/docs-site/src/content/docs/serve/recipes/index.mdx b/docs-site/src/content/docs/serve/recipes/index.mdx index fd039a5b..b0ea59d6 100644 --- a/docs-site/src/content/docs/serve/recipes/index.mdx +++ b/docs-site/src/content/docs/serve/recipes/index.mdx @@ -32,12 +32,14 @@ Codex, Claude Code and opencode don't know how big a model called `pooled` is. W |---|---|---|---|---| | Qwen3 1.7B | 16384 | 13107 | 4096 | 4096 | | Qwen3 1.7B, room short of memory for 16K | 8192 | 6553 | 2048 | 2048 | -| Qwen3.8 27B | 16384 | 13107 | 4096 | 4096 | -| Qwen3.8 27B, `?ctx=32768` | 32768 | 26214 | 8192 | 8192 | -| Qwen3.6 35B MoE | 32768 | 26214 | 8192 | 8192 | -| Qwen3.6 35B MoE, `?ctx=65536` | 65536 | 52428 | 16384 | 8192 | - -The host sets the context with `?ctx=` on the room page before starting the model. Use the MoE or the 27B at 32K for agents. The 1.7B is too small. +| Qwen3.8 27B at 16K | 16384 | 13107 | 4096 | 4096 | +| Qwen3.8 27B at 32K | 32768 | 26214 | 8192 | 8192 | +| Qwen3.8 27B at 64K | 65536 | 52428 | 16384 | 8192 | +| Qwen3.6 35B MoE at 32K | 32768 | 26214 | 8192 | 8192 | +| Qwen3.6 35B MoE at 64K | 65536 | 52428 | 16384 | 8192 | +| Qwen3.6 35B MoE at 128K | 131072 | 104857 | 32000 | 8192 | + +A room opens the 27B and the MoE at the longest context its memory holds (see [Longer context](/docs/models#longer-context)); the host can pick another in the picker's **Context** row or with `?ctx=` on the room page before starting the model. Use the MoE or the 27B at 32K or more for agents. The 1.7B is too small. ## Codex CLI diff --git a/docs-site/src/content/docs/terminal.mdx b/docs-site/src/content/docs/terminal.mdx index f01f8c03..d0893696 100644 --- a/docs-site/src/content/docs/terminal.mdx +++ b/docs-site/src/content/docs/terminal.mdx @@ -426,9 +426,9 @@ Usage More options --code the room code to use (default: a random one of six characters) - --ctx the context to ask for, in tokens (default: the longest the model takes in a - room: 16k on the 1.7B, 8k when the room is short of memory for 16k; 64k on - the 27B, 128k on the MoE) + --ctx the context to ask for, in tokens (default: the longest the room holds at full + speed: up to 64k on the 27B and 128k on the MoE, 16k on the 1.7B; shorter + when the room is short of memory) --devices the same as --wait (its older name) --signal PeerJS signaling server(s), as pooled join --no-check skip the test allocation that confirms the memory is there diff --git a/docs-site/src/content/docs/troubleshooting.mdx b/docs-site/src/content/docs/troubleshooting.mdx index d90a4b6a..3edb0263 100644 --- a/docs-site/src/content/docs/troubleshooting.mdx +++ b/docs-site/src/content/docs/troubleshooting.mdx @@ -161,7 +161,7 @@ The first answer is a bit slower than later ones. See [Networks and work Wi-Fi]( ### The answer stops with "context full" -The conversation filled the model's context. Start a new chat, or have the host reopen the room with a longer context, for example `https://pooled.run/room?ctx=65536`. See [Pick a model](/docs/models). +The conversation filled the model's context. Start a new chat, or have the host pick a longer one in **Room settings** (the model loads again), or reopen the room with one, for example `https://pooled.run/room?ctx=65536`. A room opens the 27B and the MoE at the longest context its memory holds, so a longer one may need another device. See [Longer context](/docs/models#longer-context). ### Other messages while asking diff --git a/docs/models.md b/docs/models.md index b635836c..909a7a00 100644 --- a/docs/models.md +++ b/docs/models.md @@ -6,8 +6,8 @@ Pooled ships no weights. Browsers fetch tensors by HTTP range request from publi | Model | File | Engine | Notes | |---|---|---|---| -| Qwen 3.8 27B | GGUF Q4_0 (~15 GB), includes the `nextn` draft layer | `Qwen35Engine` | 64 layers: 48 Gated DeltaNet + 16 attention; MTP speculation; 16K context by default, up to 64K | -| Qwen 3.6 35B MoE (Qwen3.6-35B-A3B) | GGUF Q4_0 (bartowski) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active per token (~3B of 35B); router, experts and shared expert on the GPU; 32K context by default, up to 128K | +| Qwen 3.8 27B | GGUF Q4_0 (~15 GB), includes the `nextn` draft layer | `Qwen35Engine` | 64 layers: 48 Gated DeltaNet + 16 attention; MTP speculation; a room opens it at the longest context it holds at full speed: 64K, 32K or 16K (8K when short), `?ctx=` up to 64K | +| Qwen 3.6 35B MoE (Qwen3.6-35B-A3B) | GGUF Q4_0 (bartowski) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active per token (~3B of 35B); router, experts and shared expert on the GPU; a room opens it at the longest context it holds at full speed: 128K, 64K or 32K (16K or 8K when short), `?ctx=` up to 128K | | Qwen3.5-122B-A10B MoE | GGUF Q4_0 (bartowski, two files, 72.5 GB) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active (~10B of 122B); in `pooled pull` / `list` / `host` from `@pooled/cli` 0.4.0, on the room page only with `?dev=1`; 32K context by default, up to 128K | | Qwen3.8 Flash Next (**experimental**) | GGUF Q4_0 (bartowski, three files, ~100.6 GB, of which a 28.8 GB n-gram table) | `Qwen4ExpEngine` (`engine/qwen4exp.js`, rooms via `room/flash.js`) | 125B, 512 experts, 10 active per token (~6B); most experts stay on disk and stream into a GPU cache (disk tier); context up to 64K, tested against llama.cpp; 128K in progress; from the command line, run the CLI from the repo ([cli/README.md](../cli/README.md#qwen38-flash-experimental-run-the-cli-from-the-repo)); in the room picker | | Qwen3 4B / 1.7B / 0.6B | GGUF Q4_0 / Q8_0 | `DenseEngine` | dense; 0.6B is the golden-test model | diff --git a/packages/openclaw/src/models.js b/packages/openclaw/src/models.js index 96fa838c..65229bc3 100644 --- a/packages/openclaw/src/models.js +++ b/packages/openclaw/src/models.js @@ -5,7 +5,8 @@ import { execFileSync } from "node:child_process"; import { readFileSync } from "node:fs"; import os from "node:os"; -import { MODELS, FILES, NEED_GB, CTX, maxSeqFor, roomBytes, pickCtx, ctxShortNote, ctxK } from "../../../room/models.js"; +import { MODELS, FILES, NEED_GB, CTX, maxSeqFor, ctxTop, roomBytes, pickCtx, ctxShortNote, ctxK } from "../../../room/models.js"; +import { roomCtxPlan } from "../../../room/ctxpick.js"; import { roomFit, shortNote, shortBy, gbUp } from "../../../room/plan.js"; import { pledgeGB } from "../../../room/pledge.js"; import { LOCAL } from "../../room-node/source.js"; @@ -37,19 +38,21 @@ export function recommendForOpenClaw(rows) { return row(OPENCLAW_MODEL) ? OPENCLAW_MODEL : recommendModel(rows); } -// the context a node opens a room with (room-node roomnode.js nodeCtxFor): the qwen35 models' largest -// (OpenClaw's prompts alone are 8-12k tokens), the room's default otherwise -export const nodeCtxFor = (model, ask = 0) => (ask > 0 || MODELS[model]?.kind !== "qwen35" || !CTX[model] ? maxSeqFor(model, ask) : CTX[model].max); -export const lib = { MODELS, FILES, NEED_GB, roomBytes, roomFit, shortNote, shortBy, gbUp, pledgeGB, nodeCtxFor, pickCtx, ctxShortNote }; +// the longest context a node opens a room with (room-node roomnode.js nodeCtxFor): the top of the qwen35 models' ladder +// (64k on the 27B, 128k on the MoE; the room walks it down to what it holds at full speed, room/ctxpick.js roomCtxPlan; +// OpenClaw's prompts alone are 8-12k tokens), the room's default otherwise +export const nodeCtxFor = (model, ask = 0) => (ask > 0 || MODELS[model]?.kind !== "qwen35" || !CTX[model] ? maxSeqFor(model, ask) : ctxTop(model)); +export const lib = { MODELS, FILES, NEED_GB, roomBytes, roomFit, shortNote, shortBy, gbUp, pledgeGB, nodeCtxFor, pickCtx, roomCtxPlan, maxSeqFor, ctxShortNote }; -// the context the plugin opens a room with: what was asked (clamped by room/models.js), else the -// model's largest room context (CTX: 16k on the 1.7B, 64k on the 27B, 128k on the MoE; OpenClaw's -// prompts alone are 8-12k tokens) -export const pluginCtx = (key, ask = 0) => (ask > 0 ? maxSeqFor(key, ask) : CTX[key]?.max ?? maxSeqFor(key)); +// the longest context the plugin opens a room with: what was asked (clamped by room/models.js), else the top of the +// model's ladder (CTX: 16k on the 1.7B, 64k on the 27B, 128k on the MoE; the room opens there when it holds it at full +// speed, else at a shorter step: room/ctxpick.js roomCtxPlan; OpenClaw's prompts alone are 8-12k tokens) +export const pluginCtx = (key, ask = 0) => (ask > 0 ? maxSeqFor(key, ask) : ctxTop(key)); // what the plugin asks the room node for: the context asked for, else none for a model with a fallback // context (the node opens the 1.7B at 16k, or at 8k when the room is short of memory for 16k: // room/models.js pickCtx), else the model's largest (pluginCtx) -// (and none for a model with a ladder, Qwen3.8 Flash: the node walks it down from its longest proven context) +// (and none for a model with a ladder, the 27B, the MoE and Qwen3.8 Flash: the node walks it down from its longest +// proven context) export const pluginAsk = (key, ask = 0) => (ask > 0 ? ask : CTX[key]?.fallback || CTX[key]?.ladder ? 0 : pluginCtx(key)); // OpenClaw's own instructions and tools are about 12k tokens and it keeps 4k for the answer: a room @@ -62,8 +65,11 @@ export function shortCtxNote(model, ctx, host = "the host") { if (!(Number.isInteger(ctx) && ctx > 0 && ctx < OPENCLAW_MIN_CTX)) return null; const max = CTX[model]?.max; const how = max >= OPENCLAW_MIN_CTX ? `\`pooled host ${model} --ctx ${max}\`` : `a model with a longer context (at least ${OPENCLAW_MIN_CTX / 1024}k)`; - const short = CTX[model]?.fallback === ctx && max >= OPENCLAW_MIN_CTX - ? ` (the room opens ${String(MODELS[model]?.label || model).split("·")[0].trim()} at ${ctxK(ctx)} when its memory is short for ${ctxK(max)}: a device lending more, or one more device, gets ${ctxK(max)})` : ""; + // (a context the room falls back to when its memory is short for the model's default: the 1.7B's 8k, the 27B's or the MoE's + // ladder below 16k / 32k) + const def = CTX[model]?.def; + const short = (CTX[model]?.fallback === ctx || (CTX[model]?.ladder?.includes(ctx) && ctx < def)) && def >= OPENCLAW_MIN_CTX + ? ` (the room opens ${String(MODELS[model]?.label || model).split("·")[0].trim()} at ${ctxK(ctx)} when its memory is short for ${ctxK(def)}: a device lending more, or one more device, gets ${ctxK(def)})` : ""; return `${host} runs this room with a ${ctx}-token context, but OpenClaw needs ${OPENCLAW_MIN_CTX / 1024}k: its own instructions and tools take about 12k tokens, so every answer would end in "Context overflow". Ask ${host} to open the room again with ${how}${short}.`; } // This machine's own room opened the model at its fallback context (room/models.js pickCtx): what to @@ -78,7 +84,8 @@ export function ownShortCtxNote(model, ctx, ctxNote, needGB = null) { // minNeed: { ctx, needGB } at the model's fallback context (the 1.7B: 8k), null without one export function modelInfo(key, ctxAsk = 0) { const ctx = pluginCtx(key, ctxAsk); - return { name: String(MODELS[key]?.label || key).split("·")[0].trim(), needGB: modelNeedGB(lib, key, ctx) ?? NEED_GB[key] ?? null, ctx, fileBytes: FILES[key]?.bytes || null, + // (needGB: at the context the room needs to open the way it did before the ladder, its default, unless one was asked) + return { name: String(MODELS[key]?.label || key).split("·")[0].trim(), needGB: modelNeedGB(lib, key, pluginAsk(key, ctxAsk) || (CTX[key]?.ladder ? 0 : ctx)) ?? NEED_GB[key] ?? null, ctx, fileBytes: FILES[key]?.bytes || null, minNeed: modelFallback(lib, key, pluginAsk(key, ctxAsk)) }; } @@ -118,8 +125,8 @@ export function modelChoices(dir, pledge) { r.hint = [ r.pulled ? "downloaded ✓" : `${fmtBytes(r.fileBytes)} download`, `needs about ${r.needGB} GB across the room${r.minNeed?.needGB ? ` (${r.minNeed.needGB} GB at ${Math.round(r.minNeed.ctx / 1024)}k)` : ""}`, - `${Math.round(r.ctx / 1024)}k context`, - r.fitsAlone && r.aloneCtx < r.ctx ? `fits on this machine alone at ${Math.round(r.aloneCtx / 1024)}k (OpenClaw needs ${OPENCLAW_MIN_CTX / 1024}k)` + CTX[r.key]?.ladder ? `up to ${Math.round(r.ctx / 1024)}k context` : `${Math.round(r.ctx / 1024)}k context`, + r.fitsAlone && r.aloneCtx < r.ctx ? `fits on this machine alone at ${Math.round(r.aloneCtx / 1024)}k${r.aloneCtx < OPENCLAW_MIN_CTX ? ` (OpenClaw needs ${OPENCLAW_MIN_CTX / 1024}k)` : ""}` : r.fitsAlone ? "fits on this machine alone" : "needs more devices", r.key === recommended ? "recommended for OpenClaw" : null, isSmall(r.key) ? "small: slow turns and tool loops in OpenClaw" : null, diff --git a/packages/openclaw/test/plugin_test.mjs b/packages/openclaw/test/plugin_test.mjs index 1db18b39..72ecc688 100644 --- a/packages/openclaw/test/plugin_test.mjs +++ b/packages/openclaw/test/plugin_test.mjs @@ -337,7 +337,9 @@ test("the 1.7B: the plugin asks the room node for no context (16k, 8k when short assert.equal(pluginCtx("qwen3-1.7b"), 16384, "what OpenClaw is told: 16k"); assert.equal(pluginAsk("qwen3-1.7b"), 0, "the node picks 16k, or 8k when the room is short"); assert.equal(pluginAsk("qwen3-1.7b", 16384), 16384, "an asked context stays asked"); - assert.equal(pluginAsk("qwen3.6-35b-moe"), 131072, "no fallback on the MoE: its largest"); + assert.equal(pluginAsk("qwen3.6-35b-moe"), 0, "the MoE's ladder: the node opens the longest it holds at full speed"); + assert.equal(pluginCtx("qwen3.6-35b-moe"), 131072, "up to 128k"); + assert.equal(pluginAsk("qwen3.8-27b"), 0, "the 27B's ladder too"); // one machine lending 4 GB: the room opens the 1.7B at 8k; 6 GB holds 16k const one = (gb) => roomFitNow(lib, { model: "qwen3-1.7b", devices: [{ name: "this machine", meta: { contribGB: gb, webgpu: true } }], ctxAsk: pluginAsk("qwen3-1.7b") }); assert.deepEqual([one(4).fits, one(4).ctx, one(4).fellBack], [true, 8192, true]); diff --git a/packages/openclaw/test/v1_test.mjs b/packages/openclaw/test/v1_test.mjs index 73bd0ba6..8d52be19 100644 --- a/packages/openclaw/test/v1_test.mjs +++ b/packages/openclaw/test/v1_test.mjs @@ -129,7 +129,8 @@ test("onboarding (host): the GPU's default pledge, the model list, 'don't downlo test("onboarding (host): the 35B MoE is preselected when this device holds it, else the 27B; no small-model warning for them", async () => { const models = tmp("models"); - for (const [gb, want] of [[40, "qwen3.6-35b-moe"], [22, "qwen3.8-27b"]]) { + // (22 GB holds the MoE at its 32k default, the ladder's full-speed step there: 64k; 20 GB holds it at no step, the 27B at 64k) + for (const [gb, want] of [[40, "qwen3.6-35b-moe"], [22, "qwen3.6-35b-moe"], [20, "qwen3.8-27b"]]) { deps.memoryDefaults = () => ({ mem: { kind: "unified", name: "M5", totalGB: 64 }, def: gb, max: 48, label: "M5 · 64 GB" }); const p = prompter(["host", String(gb), want, "stream", 1, true]); const res = await runSetup({ prompter: p, config: { plugins: { entries: { pooled: { config: { modelDir: models } } } } } }); diff --git a/packages/room-node/index.js b/packages/room-node/index.js index 0d5663be..2f0cc88f 100644 --- a/packages/room-node/index.js +++ b/packages/room-node/index.js @@ -12,7 +12,8 @@ export { createRoom, joinRoom, RoomNode, PREFIX, toApiRequest, eventEncoder, nodeServers, nodeCtxFor } from "./roomnode.js"; export { PROTOCOL } from "../../room/transport.js"; export { reconnectDelay } from "../../room/signal.js"; -export { MODELS, FILES, FILE_GB, NEED_GB, NEED_MIN_GB, SHAPE, PICKER, CTX, roomBytes, pickCtx, ctxChoices, ctxShortNote, ctxK, ngramOf } from "../../room/models.js"; +export { MODELS, FILES, FILE_GB, NEED_GB, NEED_MIN_GB, SHAPE, PICKER, CTX, roomBytes, pickCtx, ctxChoices, ctxShortNote, ctxK, ngramOf, maxSeqFor, ctxTop } from "../../room/models.js"; +export { roomCtxPlan } from "../../room/ctxpick.js"; // the room page's fit math (#271), so pooled host says "short" exactly when the room page would export { roomFit, shortNote, shortBy, gbUp, dealRoom } from "../../room/plan.js"; export { pledgeGB, ramGB, offloadFor, ngramFor, maxPledgeGB } from "../../room/pledge.js"; diff --git a/packages/room-node/roomnode.js b/packages/room-node/roomnode.js index 64b68c85..3c51235e 100644 --- a/packages/room-node/roomnode.js +++ b/packages/room-node/roomnode.js @@ -50,7 +50,8 @@ import { unpackWire, wireStats } from "../../room/wire.js"; import { TeleMeter, expertStoreOf } from "../../room/tele.js"; import { isPhoneMeta, roomFit, dealRoom, shortNote, specWithOffload, dealOffloads, parseForce, layersTotal } from "../../room/plan.js"; import { verifyHost, hostAllowed, startBusy, startFrom, joinOrder, START_PENDING_MS } from "../../room/authz.js"; -import { flashRoomLimits, ckptSlotsAt, MODELS, CTX, roomBytes, maxSeqFor, ctxForBinding, kvModeFor, kvForLoad, kvBytesPerLayerPos, MAX_NEW, MIN_ROOM, pickCtx, ctxShortNote, expertsOf, ngramOf, flashLayerCost, +import { roomCtxPlan, hostMarginFor } from "../../room/ctxpick.js"; +import { flashRoomLimits, ckptSlotsAt, MODELS, CTX, roomBytes, maxSeqFor, ctxTop, ctxForBinding, kvModeFor, kvForLoad, kvBytesPerLayerPos, MAX_NEW, MIN_ROOM, pickCtx, ctxShortNote, expertsOf, ngramOf, flashLayerCost, flashRoomFormat, flashReadyFields, readyCtxShort, flashDealReserve } from "../../room/models.js"; import { CkptIndex, CKPT_DEFAULTS, boundaryPin, pinPoints, cutPoints, turnPoint } from "./ckpt.js"; import { CkptDisk, modelFileId, prefixHash, roomKey } from "./ckptdisk.js"; @@ -101,7 +102,10 @@ const FROM_HOST = new Set(["ai-layers", "ai-ready-all", "ai-reset", "ai-redeal", // costs 1.8 GB more at 16k (5.6 GB in all); a room whose pledges hold it only at 8k opens it there // (room/models.js pickCtx, as the room page does), and --ctx 8192 asks for 8k. A model without a CTX // entry keeps the room's default. (qwen35: the deal lowers it to what every device can bind.) -export const nodeCtxFor = (model, ask = 0) => (ask > 0 || !CTX[model] ? maxSeqFor(model, ask) : CTX[model].max); +// The 27B and the MoE have a ladder (room/models.js CTX): this is its top, and the room walks it down to the longest step +// its devices hold, past the model's default (32K MoE, 16K 27B) only at full speed (room/ctxpick.js roomCtxPlan, as the +// room page). +export const nodeCtxFor = (model, ask = 0) => (ask > 0 || !CTX[model] ? maxSeqFor(model, ask) : ctxTop(model)); // what a host that closes its room says to the devices in it (bye {closed: 1}) export const HOST_CLOSED = "The host closed the room."; export const cleanName = (s, id) => String(s ?? id).replace(/[\u0000-\u001f\u007f<>"'`&]/g, "").trim().slice(0, 40) || String(id).slice(0, 8); @@ -1048,15 +1052,16 @@ export class RoomNode extends EventEmitter { // --ctx stays. -> { ctx, want, fits, fellBack, note } (note: what the room says when it fell back) // (Qwen3.8-Flash-Next: within the room's K / V format and the longest context its devices take, room/models.js // flashRoomLimits, as the room page's roomCtxPlan; fkv: that format, for the deal's sizes) + // One plan with the room page (room/ctxpick.js roomCtxPlan): a model with a ladder (the 27B, the MoE, Flash) opens at its + // longest step these pledges hold, past its default only at full speed; want: the longest the room may take (nodeCtxFor, + // held to the devices' binding limit: ctxForDevices) static ctxPick(modelKey, { want, ask = 0, kv = "f16", self, peers = [], shareCap = new Map() }) { const pl = [self, ...peers].map((d) => dealPledgeBytes(d.meta, shareCap.get(d.name), modelKey)); - const metas = [self, ...peers].map((d) => d.meta), lim = flashRoomLimits(modelKey, metas); - const fkv = lim.kv || (kv === "q8" ? "q8" : "f16"); - const pick = pickCtx(modelKey, { want, ask, cap: lim.cap, fitsAt: (c) => { - const rb = roomBytes(modelKey, c, fkv); - return !rb || roomFit(rb.L, pl, rb.layerCost || rb.layerBytes, rb.hostBytes, offloadFor(metas, rb.experts || rb.expertBytes, modelKey)).fits; - } }); - return { ...pick, fkv, note: pick.fellBack ? ctxShortNote(String(MODELS[modelKey]?.label || modelKey).split("·")[0].trim(), pick.ctx, pick.want) : "" }; + const metas = [self, ...peers].map((d) => d.meta); + if (!roomBytes(modelKey, want, kv)) return { ctx: want, want, fits: true, fellBack: false, fkv: kv === "q8" ? "q8" : "f16", note: "" }; + const plan = roomCtxPlan(modelKey, { metas, pledges: pl, ask, kvAsk: kv === "q8" ? "q8" : null, margin: hostMarginFor(modelKey, self.meta), cap: want }); + const { ctx, want: w, fits, fellBack } = plan; + return { ctx, want: w, fits, fellBack, fkv: plan.kv, note: fellBack ? ctxShortNote(String(MODELS[modelKey]?.label || modelKey).split("·")[0].trim(), ctx, w) : "" }; } setSplit(mode) { this.splitMode = mode === "speed" ? "speed" : "memory"; } // the checkpoint slots this room keeps: --ckpt's, held to the deal's reserve in a long Flash room (ai.ckptLimit) diff --git a/packages/room-node/test/roomnode_test.mjs b/packages/room-node/test/roomnode_test.mjs index 151167ed..78cf5f12 100644 --- a/packages/room-node/test/roomnode_test.mjs +++ b/packages/room-node/test/roomnode_test.mjs @@ -475,8 +475,18 @@ test("ctxPick: the 1.7B at 16k when the pledges hold it, 8k when they are short const asked = RoomNode.ctxPick("qwen3-1.7b", { want: nodeCtxFor("qwen3-1.7b", 16384), ask: 16384, self: self(4) }); assert.deepEqual([asked.ctx, asked.fits, asked.fellBack], [16384, false, false]); assert.deepEqual([RoomNode.ctxPick("qwen3-1.7b", { want, self: self(3) }).ctx, RoomNode.ctxPick("qwen3-1.7b", { want, self: self(3) }).fits], [8192, false]); - // the MoE has no fallback: its context stays - assert.equal(RoomNode.ctxPick("qwen3.6-35b-moe", { want: 131072, self: self(4) }).ctx, 131072); + // the MoE's ladder (room/models.js CTX, room/ctxpick.js roomCtxPlan): the longest step the pledges hold, past its 32k + // default only at full speed; short everywhere: short for its last step (8k) + const moe = (gb, extra = {}) => RoomNode.ctxPick("qwen3.6-35b-moe", { want: nodeCtxFor("qwen3.6-35b-moe"), self: { name: "pc", meta: { contribGB: gb, native: "node-dawn", ...extra } } }); + assert.equal(nodeCtxFor("qwen3.6-35b-moe"), 131072); + assert.deepEqual([moe(4).ctx, moe(4).fits, moe(4).fellBack], [8192, false, false]); + assert.deepEqual([moe(21).ctx, moe(21).fits, moe(21).fellBack, moe(21).note], [32768, true, false, ""], "21 GB: the default"); + assert.equal(moe(22).ctx, 65536, "22 GB: 64k"); + assert.equal(moe(24).ctx, 131072, "24 GB: 128k"); + // a GPU that holds the MoE only with experts in RAM keeps the default: a longer context would take expert cache + assert.equal(moe(16, { offload: true, ramGB: 64 }).ctx, 32768); + // --ctx 131072 stays as asked + assert.equal(RoomNode.ctxPick("qwen3.6-35b-moe", { want: 131072, ask: 131072, self: self(16) }).ctx, 131072); }); test("close() on a host tells the room it is over (bye closed, then leaving); a device told so stops waiting for it", async () => { diff --git a/room.js b/room.js index 714ffb14..eb616913 100644 --- a/room.js +++ b/room.js @@ -27,7 +27,7 @@ import { chatRecipients } from "./room/visibility.js"; import { validateApiAsk, apiPrompt, apiRun, AnswerCache, API_LIMITS, pieceDecoder, helloMeta, withStyle, apiPrompt2, apiRun2, TurnCache, EncodeCache } from "./room/api.js"; import { tokenTexts } from "./harness/model-common.js"; import { CkptStore } from "./room/ckpt-store.js"; -import { SHAPE, CTX, MODELS, DISK_COPY, NEED_GB, NEED_MIN_GB, FILE_GB, PICKER, MAX_SEQ, MAX_NEW, MAX_NEW_THINKING, MIN_ROOM, maxSeqFor, ctxForBinding, kvBytesPerLayerPos, kvModeFor, kvForLoad, hostHeldBytes, denseKvBytesPerLayerPos, roomBytes, pickCtx, ctxK, ctxShortNote, needText, expertsOf, modelLoadOptions, ngramOf, flashLayerCost, flashTabHostMargin, +import { SHAPE, CTX, MODELS, DISK_COPY, NEED_GB, NEED_MIN_GB, FILE_GB, PICKER, MAX_SEQ, MAX_NEW, MAX_NEW_THINKING, MIN_ROOM, maxSeqFor, ctxTop, ctxFloor, needGBAt, ctxForBinding, kvBytesPerLayerPos, kvModeFor, kvForLoad, hostHeldBytes, denseKvBytesPerLayerPos, roomBytes, pickCtx, ctxK, ctxShortNote, needText, expertsOf, modelLoadOptions, ngramOf, flashLayerCost, flashTabHostMargin, flashLongMeta, flashRoomFormat, flashReadyFields, readyCtxShort, flashDealReserve } from "./room/models.js"; // the context window of the loaded engine (per model: room/models.js CTX; 2048 for the small ones) const ctxMax = () => ai.engine?.maxSeq || MAX_SEQ; @@ -599,10 +599,10 @@ function spareGBOf(meta) { const ceil = Math.min(rule.capped ? rule.max : 64, meta?.pledgeMax || 64, meta?.budgetGB ? Math.max(SMALLEST_NEED, Math.floor(meta.budgetGB)) : 64); return Math.max(0, +(ceil - pledgeGB(meta)).toFixed(1)); } -// The picker's context choice per model (Qwen3.8 Flash: room/ctxpick.js ctxOptions), remembered for this room in this +// The picker's context choice per model (the 27B, the 35B MoE, Qwen3.8 Flash: room/ctxpick.js ctxOptions), remembered for this room in this // tab (sessionStorage, per room code): the longest context the room may open at. Start deals with it (aiStart: // roomCtxPlan's choice, as roomFitFor), and the devices load the room's context from ai-load. 0: none (the longest proven -// context the room holds). ?ctx=N overrides it (an explicit ask, tried alone). +// context the room holds at full speed: roomCtxPlan). ?ctx=N overrides it (an explicit ask, tried alone). const CTX_URL_ASK = +new URLSearchParams(location.search).get("ctx") || 0; const CTX_PICK_KEY = "pooled-ctx"; function ctxChoiceFor(key) { @@ -639,7 +639,7 @@ function roomFitFor(key) { const pick = plan; const fit = fitAt(pick.ctx), spare = devs.map((d) => spareGBOf(d.meta)); const out = { fit, devs, ctx: pick.ctx, want: pick.want, fellBack: pick.fellBack, kv, - // the picker's context choices (Flash): each one's need here, and why it is off (room/ctxpick.js ctxOptions) + // the picker's context choices (the 27B, the MoE, Flash): each one's need here, and why it is off (room/ctxpick.js ctxOptions) ctxOpts: ask ? [] : ctxOptions(key, { metas, pledges, margin, kvAsk }), shortGB: gbUp(shortBy(fit, spare)), note: shortNote(shortName(key), fit, devs.map((d) => d.name), spare, PAGE_SHORT_FIX) }; // a model kept on disk (room/models.js DISK_COPY): its need is the GPU memory the planner itself needs from these @@ -690,7 +690,9 @@ function renderLadder(pledged, rows = fitLadder(pledged)) { const html = (none ? '

Needs a device with WebGPU

' : "") + rows.map((x) => { // fits only at the model's fallback context: say which, with what it needs there const back = x.ok && x.f?.fellBack; - const gb = `${back ? `${NEED_MIN_GB[x.key] ?? ""} GB · ${ctxK(x.f.ctx)}` : `${NEED_GB[x.key] ?? ""} GB`}`; + // (at the ladder's last step or the fallback: NEED_MIN_GB; at a step between it and the default: the deal's own need there) + const backGB = back && (x.f.ctx === ctxFloor(x.key) ? NEED_MIN_GB[x.key] : needGBAt(x.key, x.f.ctx)); + const gb = `${back ? `${backGB ?? ""} GB · ${ctxK(x.f.ctx)}` : `${NEED_GB[x.key] ?? ""} GB`}`; const fig = x.ok ? `${gb}fits` : none ? gb : `${x.short} GB short`; // with no device that can hold layers, nothing reads as picked: there is nothing to start yet const sel = !none && x.key === $("ai-model").value; @@ -713,7 +715,7 @@ function renderLadder(pledged, rows = fitLadder(pledged)) { if (had) el.querySelector('.rung[tabindex="0"]')?.focus({ preventScroll: true }); } function pickRung(b) { if (!b || $("ai-model").disabled) return; setModelValue(b.dataset.k); modelTouched = true; updateCluster(); } -// The context choice under the rows (a model with choices: Qwen3.8 Flash, room/ctxpick.js ctxOptions): each proven length +// The context choice under the rows (a model with choices: the 27B, the 35B MoE, Qwen3.8 Flash, room/ctxpick.js ctxOptions): each proven length // with the GPU memory it needs in this room and about how long reading a full paste takes there; one this room can't // hold is off and says why. The picked one is the context the room opens at (roomFitFor's: the choice, or the longest // that fits). The same numbers as the need line, the pool and Start (roomCtxPlan). Room settings has the same choice @@ -3855,11 +3857,13 @@ async function aiStart(modelArg) { ai.lapStat = null; // a new chain: lap timeouts start from the fixed fallbacks again const modelKey = $("ai-model").value; const M = MODELS[modelKey]; - // context for this room: the model's default, or ?ctx=N up to its cap (room/models.js CTX); every - // device builds its engine with it. Without ?ctx= a room whose pledges hold the model only at its - // fallback (the 1.7B: 8K for 16K) opens it there (pickCtx, below) + // context for this room: ?ctx=N up to the model's cap (room/models.js CTX), else the picker's choice, else the longest + // step of the model's ladder the room holds (past its default only at full speed: room/ctxpick.js roomCtxPlan, below), + // or its fallback (the 1.7B: 8K for 16K); every device builds its engine with it const CTX_ASK = CTX_URL_ASK; - let ROOM_CTX = maxSeqFor(modelKey, CTX_ASK); + // (without ?ctx=: the top of the model's ladder, ctxTop; roomCtxPlan below walks it down to what the room holds) + let ROOM_CTX = CTX_ASK > 0 ? maxSeqFor(modelKey, CTX_ASK) : ctxTop(modelKey); + const ROOM_CTX0 = ROOM_CTX; const ROOM_KV = kvModeFor(modelKey, KV_ASK); // KV cache format for every device: f16, or int8 with ?kv=q8 // a model host that is not the room's creator (biggestPeerId) has links only to the creator and to // devices it dialed: link every device first, so the deal, ai-layers and ai-ready-all reach them all @@ -3944,7 +3948,7 @@ async function aiStart(modelArg) { // (a device that refused the offload it was dealt, ai.noDisk: a tab keeping the model off its disk, an older node, // counts without any offload: no disk tier, nothing parked) const metasNow = () => [myMeta, ...ai.chain.map((id) => { const m = conns.get(id)?.meta; return m && ai.noDisk?.has(nameOf(id)) ? { ...m, tabDisk: undefined, offload: undefined, ramGB: undefined } : m; })]; - const ctxCap = ROOM_CTX < maxSeqFor(modelKey, CTX_ASK) ? ROOM_CTX : Infinity; // (a qwen35 room's binding limit) + const ctxCap = ROOM_CTX < ROOM_CTX0 ? ROOM_CTX : Infinity; // (a qwen35 room's binding limit) let roomKv = null; const planNow = () => { const metas = metasNow(); diff --git a/room/ctxpick.js b/room/ctxpick.js index 6c167980..d1940513 100644 --- a/room/ctxpick.js +++ b/room/ctxpick.js @@ -6,8 +6,8 @@ // Qwen3.8-Flash-Next (rooms long-context PR 5): the room's K / V format and the longest context its devices take // (room/models.js flashRoomLimits), the ladder of proven lengths (CTX ladder), and the user's context choice in the // picker (want: the longest it may open at; the room walks the ladder down from there when its memory is short). -import { MODELS, CTX, FLASH_CTX_OPTIONS, maxSeqFor, kvModeFor, pickCtx, roomBytes, flashRoomLimits, flashTabHostMargin, ctxK } from "./models.js"; -import { roomFit, dealRoom, dealWithHostMargin, minFitPledge, gbUp } from "./plan.js"; +import { MODELS, CTX, maxSeqFor, ctxTop, kvModeFor, pickCtx, roomBytes, flashRoomLimits, flashTabHostMargin, ctxK } from "./models.js"; +import { roomFit, dealRoom, dealWithHostMargin, minFitPledge, gbUp, isPhoneMeta } from "./plan.js"; import { offloadFor } from "./pledge.js"; // costs(ctx, kv) -> { L, layers (a number: one layer's bytes, or Flash's per-layer costs), hostBytes, experts }: the @@ -36,7 +36,15 @@ export function fitAtCtx(model, { c, kv, pledges, metas, costs, margin = null, d return { fit: d.fit, deal: d, costs: x, off }; } -// The room's context for `model` and its fit there. +// The room's context for `model` and its fit there: the picker (room.js roomFitFor), Start (room.js aiStart), the room +// node (packages/room-node RoomNode.ctxPick: `pooled host`, `pooled serve`'s room, the OpenClaw plugin) and `pooled host`'s +// screen (cli/lib/hostui.js roomFitNow) all call this, so they open a room at the same context. +// Without an ask or a choice a model with a ladder (CTX: the 27B, the 35B MoE, Flash) opens at the longest step the room +// holds, but past its default (CTX def: 16K on the 27B, 32K on the MoE) only at full speed (fastAt): the room holds that +// context with no expert offloaded (fit.offload: each token then copies experts from RAM or disk) and on no more devices +// than at the default (a longer context that pulls a phone, or one more hop, into the chain slows every token). Decode +// reads only the filled positions (engine attn_dec splits follow the length), so a longer window alone costs memory, not +// speed. Such a room keeps the default, and the picker's context row has the longer ones a click away. // metas / pledges: the devices that may be dealt layers, host first (pledges in bytes: room/pledge.js dealPledgeBytes); // ask: an explicit context (?ctx=, --ctx: tried alone); choice: the picker's context choice (the longest the room opens // at; the ladder below it when short); kvAsk: the room's ?kv= (Qwen 3.5 only); costs / margin / deal: as fitAtCtx. @@ -47,22 +55,27 @@ export function fitAtCtx(model, { c, kv, pledges, metas, costs, margin = null, d export function roomCtxPlan(model, { metas, pledges, ask = 0, choice = 0, kvAsk = null, costs = roomCosts(model), margin = null, cap = Infinity, wrapOff }) { const lim = flashRoomLimits(model, metas); const kv = lim.kv || kvModeFor(model, kvAsk), top = Math.min(lim.cap, cap); - const want = ask > 0 ? maxSeqFor(model, ask) : choice > 0 ? Math.min(maxSeqFor(model), maxSeqFor(model, choice)) : maxSeqFor(model); + const want = ask > 0 ? maxSeqFor(model, ask) : choice > 0 ? Math.min(ctxTop(model), maxSeqFor(model, choice)) : ctxTop(model); const fitAt = (c, pl = pledges, deal = {}) => fitAtCtx(model, { c, kv, pledges: pl, metas, costs, margin, deal, wrapOff }); - const pick = pickCtx(model, { want, ask, cap: top, fitsAt: (c) => !!fitAt(c)?.fit.fits }); + // full speed at c: the deal by memory (phones only when the computers can't hold it, as Start and the node deal) holds + // every expert in GPU memory, on at most as many devices as at the default + let usedDef; + const used = (c) => { const r = fitAt(c, pledges, { phone: metas.map(isPhoneMeta) }); return r?.fit.fits && !r.fit.offload ? r.deal.used.length : -1; }; + const fastAt = ask > 0 || choice > 0 ? null : (c) => { usedDef ??= used(Math.min(CTX[model]?.def ?? c, top)); const n = usedDef > 0 ? used(c) : -1; return n > 0 && n <= usedDef; }; + const pick = pickCtx(model, { want, ask, cap: top, fitsAt: (c) => !!fitAt(c)?.fit.fits, fastAt }); return { ...pick, kv, cap: top, fitAt }; } -// The picker's context choices for a model (Qwen3.8-Flash-Next: FLASH_CTX_OPTIONS, proven lengths only), each with +// The picker's context choices for a model (CTX options: the 27B, the 35B MoE, Flash; proven lengths only), each with // what it needs in this room: [{ ctx, label, fits, needGB, shortGB, why, capped }] (needGB: the GPU memory the planner // needs from these devices at that context, minFitPledge, as the picker's need line; shortGB / why: how much more, when // it doesn't fit; capped: a device here can't take that long a context). [] for a model without choices. export function ctxOptions(model, { metas, pledges, costs = roomCosts(model), margin = null, kvAsk = null }) { - if (!CTX[model]?.ladder) return []; + if (!CTX[model]?.options) return []; const lim = flashRoomLimits(model, metas); const kv = lim.kv || kvModeFor(model, kvAsk); const has = pledges.reduce((a, b) => a + (b > 0 ? b : 0), 0) / 2 ** 30; - return FLASH_CTX_OPTIONS.map((c) => { + return CTX[model].options.map((c) => { const capped = c > lim.cap; const fitAt = (pl) => fitAtCtx(model, { c, kv, pledges: pl, metas, costs, margin }); const f = capped ? null : fitAt(pledges); diff --git a/room/models.js b/room/models.js index 3215d974..4641aaec 100644 --- a/room/models.js +++ b/room/models.js @@ -4,7 +4,8 @@ import { qwen4expConfig, qwen4expShardBytes } from "../engine/gguf.js"; import { OFFLOAD_CKPT_SLOTS } from "./plan.js"; // GB each model needs at its default context (the picker's "needs N GB"); NEED_MIN_GB: what a model -// with a fallback context (CTX[model].fallback) needs there, the least a room can start it with +// with a fallback context (CTX[model].fallback, or its ladder's last step: ctxFloor) needs there, the least a room can +// start it with (the 27B's and the MoE's are set at the end of this file, from roomBytes) export const NEED_GB = { "qwen3-0.6b": 0.8, "qwen3-1.7b": 5.6, "qwen3-4b": 4.6, "qwen3.8-27b": 17.0, "qwen3.6-35b-moe": 22.5, "qwen3.5-122b-moe": 72.0, // Qwen3.8-Flash-Next: its 48 layers with their state at the default context, the embedding and the head, from the // planner's own costs (roomBytes at CTX def: set at the end of this file, flashLayerCost), in GiB (2^30, the unit the @@ -462,7 +463,7 @@ export const FLASH_CTX_PROVEN = [2048, 8192, 16384, 32768, 65536]; export const FLASH_CTX_OPTIONS = [8192, 32768, 65536, 131072].filter((c) => FLASH_CTX_PROVEN.includes(c)); function FLASH_CTX_ENTRY() { const top = Math.max(...FLASH_CTX_PROVEN); - return { def: top, max: top, ladder: FLASH_CTX_PROVEN.slice().sort((a, b) => b - a) }; + return { def: top, max: top, ladder: FLASH_CTX_PROVEN.slice().sort((a, b) => b - a), options: FLASH_CTX_OPTIONS }; } // Context per model, in tokens: the default a room opens with and the most ?ctx=N may ask for, and // for some a fallback: the context a room opens with instead when its pledges can't hold the model @@ -477,9 +478,26 @@ function FLASH_CTX_ENTRY() { // when every device can bind it (ctxForBinding); its cap stays at 64K (128 MiB) until 128K is // checked on it. The practical limit is prefill speed, not memory // (docs/long-context-and-sessions.md: needle and speed at 32K..128K). +// +// A ladder (rooms long-context PR 5 for Flash; the 27B and the 35B MoE since this ladder's PR): the lengths a room may open +// the model at, longest first. A room opens at the longest step its devices hold (roomCtxPlan: pickCtx), down to the +// shortest; above `def` (the context the model's need, NEED_GB, is quoted at, and what rooms opened at before the ladder) +// only when the longer context costs no speed: the room holds it with every expert in GPU memory and on no more devices +// than at `def` (room/ctxpick.js roomCtxPlan fastAt; decode itself reads only the filled positions, so a longer window +// alone is no slower). `options`: the picker's context row (room/ctxpick.js ctxOptions). Only lengths checked correct +// on that model are on its ladder (docs/long-context-and-sessions.md): +// 27B 64K: needle found (tests/needle_ctx.js, 2026-09-30); 16K and 32K: needle found (attn decode v2, bench-log +// 2026-10-01); 8K: what its rooms ran before 16K (MAX_SEQ_LONG), and the same engine as at 16K / 32K (below). +// 128K is not checked (256 MiB per K/V buffer, the 128K prefill was stopped): not offered. +// MoE 32K, 64K, 96K, 128K: needle found (tests/needle_ctx.js, 2026-09-30), spec = plain, no NaN; tests/test_moe.js +// MATCHes llama.cpp with CTX=131072 (this PR); 8K and 16K: the engine at <= 32K positions is the 32K engine (one +// 256-position split length, engine/qwen35.js faSplit; decode splits follow the filled length), so it gives the +// 32K run's tokens on any prompt that fits. +export const CTX_LADDER_27B = [65536, 32768, 16384, 8192]; +export const CTX_LADDER_MOE = [131072, 65536, 32768, 16384, 8192]; export const CTX = { - "qwen3.8-27b": { def: 16384, max: 65536 }, - "qwen3.6-35b-moe": { def: 32768, max: 131072 }, + "qwen3.8-27b": { def: 16384, max: 65536, ladder: CTX_LADDER_27B, options: [8192, 16384, 32768, 65536] }, + "qwen3.6-35b-moe": { def: 32768, max: 131072, ladder: CTX_LADDER_MOE, options: [8192, 32768, 65536, 131072] }, // the 122B MoE: 12 attention layers, 2 KV heads x 256 (the 35B's K/V row): 24 KB per position in f16, 128 MiB per // layer's K at 131072. Checked at short contexts so far (phase 1): its default stays at the 35B's "qwen3.5-122b-moe": { def: 32768, max: 131072 }, @@ -591,6 +609,14 @@ export const ctxForBinding = (meta, ctx, kv = "f16", bindBytes = WEBGPU_MIN_BIND if (!per) return ctx; return Math.min(ctx, Math.floor(Math.max(bindBytes, WEBGPU_MIN_BIND) / per / 256) * 256); }; +// The longest context a room opens `model` at without an ask: the top of its ladder (27B 64K, MoE 128K, Flash 64K), else its +// default (the 1.7B: 16K). Rooms open there only when their memory holds it at full speed (roomCtxPlan). +export const ctxTop = (model) => CTX[model]?.ladder?.[0] ?? maxSeqFor(model); +// The shortest context a room opens `model` at (its fallback, or its ladder's last step), or null without one below def +export const ctxFloor = (model) => { + const c = CTX[model], lo = c?.fallback ?? c?.ladder?.[c.ladder.length - 1]; + return lo > 0 && lo < c.def ? lo : null; +}; export const maxSeqFor = (model, ask = 0) => { const c = CTX[model]; if (!c) return MODELS[model]?.kind === "qwen35" ? MAX_SEQ_LONG : MAX_SEQ; @@ -610,12 +636,18 @@ export const ctxChoices = (model, want = maxSeqFor(model), ask = 0) => { // -> { ctx, want, fits, fellBack } (fellBack: it fits, but only at a shorter context than `want`) // cap: the longest context the room's devices take (flashRoomLimits): longer choices are skipped, and when every choice // is longer the room is tried at the cap alone -export function pickCtx(model, { want = maxSeqFor(model), ask = 0, fitsAt, cap = Infinity }) { +// fastAt(ctx) -> bool (room/ctxpick.js roomCtxPlan, a room opened without a choice): a step longer than the model's def is +// taken only when fastAt says the room runs it at full speed there; want and fellBack are then measured against def (a +// room at 32K of the MoE's 128K ladder did not fall back: 32K is its default; one at 16K did) +export function pickCtx(model, { want = maxSeqFor(model), ask = 0, fitsAt, cap = Infinity, fastAt = null }) { let cs = ctxChoices(model, want, ask).filter((c) => c <= cap); if (!cs.length) cs = [Math.min(want, cap)]; - const i = cs.findIndex((c) => !!fitsAt(c)); + const def = CTX[model]?.def ?? want; + const ok = (c) => !!fitsAt(c) && (!fastAt || c <= def || !!fastAt(c)); + const i = cs.findIndex(ok); const ctx = cs[i < 0 ? cs.length - 1 : i]; - return { ctx, want, fits: i >= 0, fellBack: i > 0 }; + const aim = fastAt && !(ask > 0) ? Math.min(want, def) : want; + return { ctx, want: aim, fits: i >= 0, fellBack: i >= 0 && (fastAt ? ctx < aim : i > 0) }; } // "16K", "8K": a context in words export const ctxK = (n) => `${Math.round(n / 1024)}K`; @@ -624,7 +656,7 @@ export const ctxK = (n) => `${Math.round(n / 1024)}K`; export const ctxShortNote = (label, ctx, want) => `${label} · ${ctxK(ctx)} context: the room's memory is short for ${ctxK(want)}`; // "needs 5.6 GB (4 GB at 8K)": a model's need at its default, and at its fallback when it has one export const needText = (model, gb = NEED_GB[model]) => { - const fb = CTX[model]?.fallback, min = NEED_MIN_GB[model]; + const fb = ctxFloor(model), min = NEED_MIN_GB[model]; return fb && min != null ? `${gb} GB (${min} GB at ${ctxK(fb)})` : `${gb} GB`; }; @@ -672,3 +704,12 @@ export const MIN_ROOM = 32; // a prompt must leave at least this many tokens // Flash's "needs N GB": the planner's own bytes at its default context (roomBytes: every layer's cost, the host's part), // rounded up to a tenth, so the picker's number follows the memory model instead of a fixed figure NEED_GB["qwen3.8-flash"] = (() => { const rb = roomBytes("qwen3.8-flash", CTX["qwen3.8-flash"].def); return Math.ceil((rb.L * rb.layerBytes + rb.hostBytes) / 2 ** 30 * 10 - 1e-9) / 10; })(); +// What a model's whole deal needs at ctx, in GiB rounded up to a tenth (roomBytes: every layer with its KV cache there and +// what the host holds besides), or null for a model without a SHAPE: NEED_GB's own rule +export const needGBAt = (model, ctx, kv = "f16") => { + const rb = roomBytes(model, ctx, kv); + return rb ? Math.ceil((rb.L * rb.layerBytes + rb.hostBytes) / 2 ** 30 * 10 - 1e-9) / 10 : null; +}; +// the 27B and the MoE at their ladder's last step (8K): the least a room opens them with ("17 GB (16.5 GB at 8K)"), as +// NEED_GB less the KV cache 8K saves against def (NEED_GB's label sits a little above the deal's bytes; so does this) +for (const k of ["qwen3.8-27b", "qwen3.6-35b-moe"]) NEED_MIN_GB[k] = +(NEED_GB[k] - (needGBAt(k, CTX[k].def) - needGBAt(k, ctxFloor(k)))).toFixed(1); diff --git a/tests/test_moe.js b/tests/test_moe.js index 5e2506f1..925b43a6 100644 --- a/tests/test_moe.js +++ b/tests/test_moe.js @@ -27,7 +27,8 @@ let t0 = performance.now(); const experts = await offloadFromEnv(device, L, model); const weights = BIG ? await streamWeights(model, device, { lo: 0, hi: L, hasEmbed: true, hasHead: true, mtp: hasMtp, experts }, (i) => { if (i % 8 === 7) console.log(` layer ${i + 1}/${L} on the GPU, ${((performance.now() - t0) / 1000).toFixed(0)} s`); }) : await model.weights({ lo: 0, hi: L, hasEmbed: true, hasHead: true, mtp: hasMtp, experts }); -const eng = await Qwen35Engine.create({ device, meta: G.meta, weights, layerRange: [0, L], hasEmbed: true, hasHead: true, maxSeq: 512, +// CTX=N: the engine's context (maxSeq; default 512): CTX=131072 checks the goldens in an engine sized as a 128K room's +const eng = await Qwen35Engine.create({ device, meta: G.meta, weights, layerRange: [0, L], hasEmbed: true, hasHead: true, maxSeq: +(Deno.env.get("CTX") || 512), // DRAFTCHAIN=0 / SPECFUSE=0: per-submit drafts / separate verify submits (A/B; same output) draftChain: Deno.env.get("DRAFTCHAIN") !== "0", specFuse: Deno.env.get("SPECFUSE") !== "0", moeNormRouter: Deno.env.get("MOE_NORM_ROUTER") !== "0", // MOE_NORM_ROUTER=0: rmsnorm + router GEMV launches instead of moe_nrt (A/B) diff --git a/tests/unit/ctxpick_test.js b/tests/unit/ctxpick_test.js index 57d5d425..6e69c029 100644 --- a/tests/unit/ctxpick_test.js +++ b/tests/unit/ctxpick_test.js @@ -125,3 +125,109 @@ Deno.test("ctxpick: the demo room (Mac tab 22 + PC 12) holds 128K by the memory const old = [tab(22, { tabDisk: { ramGB: 3, models: [F] } }), metas[1]]; ok(!fitAtCtx(F, { c: 131072, kv: "f16", pledges, metas: old, costs: roomCosts(F), margin: hostMarginFor(F, old[0]) }).fit.fits); }); + +// ---- the 27B and the 35B MoE: a ladder too (room/models.js CTX ladder / options), the default past CTX def only at full speed +import { SHAPE, roomBytes, kvBytesPerLayerPos, hostHeldBytes, ctxTop, NEED_GB, NEED_MIN_GB, needText } from "../../room/models.js"; +import { isPhoneMeta } from "../../room/plan.js"; +const Q35 = ["qwen3.8-27b", "qwen3.6-35b-moe"]; +// Start's costs for a Qwen 3.5 model, as room.js aiStart builds them from the header (layers 0-3's average weights plus +// the attention layers' K / V at the context, kvBytesPerLayerPos on the header's meta; the embedding, the head and the +// draft block on the host): here from SHAPE's header numbers and a meta with the model's attention shape +const qMeta = (m) => ({ "qwen35.attention.head_count_kv": m === "qwen3.8-27b" ? 4 : 2, "qwen35.attention.key_length": 256, "qwen35.full_attention_interval": 4 }); +const q35Start = (m) => { + const s = SHAPE[m], host = hostHeldBytes("qwen35", { embed: s.embed, out: s.out, mtp: s.mtp }), ex = roomBytes(m, 8192).experts; + return (c, kv) => ({ L: s.L, layers: s.layer + c * kvBytesPerLayerPos(qMeta(m), kv), hostBytes: host, experts: ex }); +}; +const t = (gb, x = {}) => ({ webgpu: true, contribGB: gb, ...x }); +const iphone = () => ({ webgpu: true, contribGB: 1, ua: "iPhone" }); +const rtx = (gb, ram) => ({ webgpu: true, contribGB: gb, native: "node-dawn", ...(ram ? { offload: true, ramGB: ram } : {}) }); +const QROOMS = { + "Mac 16 + desktop 7 + iPhone (24 GB)": [t(16), t(7), iphone()], + "Mac 17 + desktop 6 + iPhone (24 GB)": [t(17), t(6), iphone()], + "Mac 14 + desktop 8 + iPhone (23 GB)": [t(14), t(8), iphone()], + "Mac 18 + desktop 4 (22 GB)": [t(18), t(4)], + "two 16 GB laptops": [t(16), t(16)], + "a GB10 lending 40": [rtx(40)], + "an RTX 12 GB with 48 GB of RAM": [rtx(12, 48)], + "an RTX 12 GB with RAM + a Mac 12": [rtx(12, 48), t(12)], + "a 16 GB laptop alone": [t(16)], + "a 20 GB laptop alone": [t(20)], +}; +const qPledges = (metas, m) => metas.map((x) => dealPledgeBytes(x, Infinity, m)); + +Deno.test("ctxpick (27B, MoE): the picker and Start pick the same context and fit, for every room and every choice", () => { + eq(CTX["qwen3.8-27b"].options, [8192, 16384, 32768, 65536]); + eq(CTX["qwen3.6-35b-moe"].options, [8192, 32768, 65536, 131072]); + for (const m of Q35) for (const [name, metas] of Object.entries(QROOMS)) for (const choice of [0, ...CTX[m].options]) { + const pledges = qPledges(metas, m), at = `${m}, ${name}, choice ${choice}`; + const picker = roomCtxPlan(m, { metas, pledges, choice }), start = roomCtxPlan(m, { metas, pledges, choice, costs: q35Start(m) }); + eq([start.ctx, start.want, start.kv, start.fits, start.fellBack], [picker.ctx, picker.want, picker.kv, picker.fits, picker.fellBack], at); + ok(CTX[m].ladder.includes(picker.ctx), `${at}: ${picker.ctx} is on the ladder`); + if (choice) ok(picker.ctx <= choice, `${at}: never past the choice`); + const pf = picker.fitAt(picker.ctx).fit, sf = start.fitAt(start.ctx).fit; + eq([sf.fits, sf.caps, !!sf.offload], [pf.fits, pf.caps, !!pf.offload], `${at}: the fit`); + const need = (p) => minFitPledge(pledges, (pl) => !!p.fitAt(p.ctx, pl).fit.fits); + const np = need(picker), ns = need(start); + eq(np && gbUp(np.bytes), ns && gbUp(ns.bytes), `${at}: the need`); + if (pf.fits) { + const d = start.fitAt(start.ctx, pledges, { mode: "memory", phone: metas.map(isPhoneMeta) }).deal; + ok(d.fit.fits, `${at}: Start's deal fits`); + d.used.forEach((i, k) => ok(d.held[k] <= pledges[i] + 1, `${at}: device ${i} holds ${d.held[k] / GB} of ${pledges[i] / GB}`)); + } + } +}); + +Deno.test("ctxpick (27B, MoE): each context choice's need is Start's at that choice; one that fits is where Start opens", () => { + for (const m of Q35) for (const [name, metas] of Object.entries(QROOMS)) { + const pledges = qPledges(metas, m), opts = ctxOptions(m, { metas, pledges }); + eq(opts.map((o) => o.ctx), CTX[m].options, name); + for (const o of opts) { + const s = roomCtxPlan(m, { metas, pledges, choice: o.ctx, costs: q35Start(m) }); + const mm = minFitPledge(pledges, (pl) => !!fitAtCtx(m, { c: o.ctx, kv: s.kv, pledges: pl, metas, costs: q35Start(m) }).fit.fits); + eq(o.needGB, mm ? gbUp(mm.bytes) : null, `${m} ${name} ${o.label}: need`); + ok(!o.capped, `${m} ${name} ${o.label}: no device caps a qwen35 room in the picker`); + if (o.fits) eq(s.ctx, o.ctx, `${m} ${name} ${o.label}: Start opens there`); + else ok(!s.fits || s.ctx < o.ctx, `${m} ${name} ${o.label}`); + } + } +}); + +Deno.test("ctxpick (27B, MoE): no choice opens at the longest step held at full speed, never below where rooms opened before", () => { + const plan = (m, name) => { const metas = QROOMS[name]; return roomCtxPlan(m, { metas, pledges: qPledges(metas, m) }); }; + const M = "qwen3.6-35b-moe", D = "qwen3.8-27b"; + eq([ctxTop(M), ctxTop(D), maxSeqFor(M), maxSeqFor(D)], [131072, 65536, 32768, 16384]); + // Mac + desktop + iPhone, 24 GB: the computers hold the MoE at 128K (the phone stays out), so it opens there + const a = plan(M, "Mac 16 + desktop 7 + iPhone (24 GB)"); + eq([a.ctx, a.fits, a.fellBack, a.want], [131072, true, false, 32768]); + eq(a.fitAt(a.ctx, qPledges(QROOMS["Mac 16 + desktop 7 + iPhone (24 GB)"], M), { phone: [false, false, true] }).deal.used, [0, 1]); + // the same 24 GB split 17 / 6: 128K would need the phone to hold layers (a slow hop every token): 64K, 128K a click away + const b = plan(M, "Mac 17 + desktop 6 + iPhone (24 GB)"); + eq([b.ctx, b.fellBack], [65536, false]); + const bm = QROOMS["Mac 17 + desktop 6 + iPhone (24 GB)"], b128 = roomCtxPlan(M, { metas: bm, pledges: qPledges(bm, M), choice: 131072 }); + eq(b128.ctx, 131072, "chosen in the picker: 128K"); + ok(b128.fitAt(131072, qPledges(bm, M), { phone: bm.map(isPhoneMeta) }).deal.used.includes(2), "with the phone holding layers"); + // a GPU that holds the MoE only with experts in RAM keeps the default: a longer context would take expert cache + const o = plan(M, "an RTX 12 GB with 48 GB of RAM"); + eq([o.ctx, o.fits, !!o.fitAt(o.ctx).fit.offload], [32768, true, true]); + // the 27B: 64K wherever the same devices hold it; a 16 GB laptop alone holds no step (17 GB at 16K, 16.5 at 8K) + eq(plan(D, "Mac 16 + desktop 7 + iPhone (24 GB)").ctx, 65536); + eq(plan(D, "a 20 GB laptop alone").ctx, 65536); + const lone = plan(D, "a 16 GB laptop alone"); + eq([lone.ctx, lone.fits], [8192, false], "short everywhere: short for the ladder's last step"); + // every room: where it opened before the ladder (CTX def), it opens at that or longer, the same fit + for (const m of Q35) for (const [name, metas] of Object.entries(QROOMS)) { + const pledges = qPledges(metas, m), p = roomCtxPlan(m, { metas, pledges }); + const before = p.fitAt(CTX[m].def).fit; + if (before.fits) ok(p.fits && p.ctx >= CTX[m].def && !p.fellBack, `${m} ${name}: ${p.ctx}`); + // past the default only with every expert on a GPU and no more devices than at the default + if (p.ctx > CTX[m].def) { + const used = (c) => p.fitAt(c, pledges, { phone: metas.map(isPhoneMeta) }).deal; + ok(!used(p.ctx).fit.offload && used(p.ctx).used.length <= used(CTX[m].def).used.length, `${m} ${name}`); + } + } + // ?ctx= stays as asked (tried alone) + eq(roomCtxPlan(M, { metas: QROOMS["a 16 GB laptop alone"], pledges: qPledges(QROOMS["a 16 GB laptop alone"], M), ask: 131072 }).ctx, 131072); + // the picker's need words: the default's need, and the ladder's last step's + eq(needText(D), `${NEED_GB[D]} GB (${NEED_MIN_GB[D]} GB at 8K)`); + ok(NEED_MIN_GB[M] < NEED_GB[M]); +}); diff --git a/tests/unit/flash_room_test.js b/tests/unit/flash_room_test.js index e9f3ae02..ffb0dd4a 100644 --- a/tests/unit/flash_room_test.js +++ b/tests/unit/flash_room_test.js @@ -36,7 +36,7 @@ Deno.test("flash catalogue: dev-gated entry, pinned files, shape from the GGUF h // context: dense attention is exact to 2051 cells and the phase-1 engine refuses more // the longest proven context is the default and the cap; the ladder walks the proven lengths down (rooms long-context PR 5) eq([CTX[KEY], maxSeqFor(KEY), maxSeqFor(KEY, 32768), kvModeFor(KEY, "q8")], - [{ def: 65536, max: 65536, ladder: [65536, 32768, 16384, 8192, 2048] }, 65536, 32768, "f16"]); + [{ def: 65536, max: 65536, ladder: [65536, 32768, 16384, 8192, 2048], options: [8192, 32768, 65536] }, 65536, 32768, "f16"]); // shape: 48 layers, 512 experts with layers 0-5's Q4_1 down experts parked as Q8, the 4-stream residual, the table eq([S.L, S.park.nExp, S.park.runs.map(([n]) => n), S.hDim, wireWidth(KEY), S.ngram, ngramOf(KEY)], [48, 512, [6, 42], 10240, 10240, { layer: 1, bytes: 28800138240 }, { layer: 1, bytes: 28800138240 }]); diff --git a/tests/unit/models_test.js b/tests/unit/models_test.js index 75d1b5b9..d2fe509c 100644 --- a/tests/unit/models_test.js +++ b/tests/unit/models_test.js @@ -279,8 +279,10 @@ Deno.test("pickCtx: 16K when it fits, 8K fallback when only that fits, an explic eq(ctxChoices(m, 16384, 16384), [16384], "an explicit ask is tried alone"); eq(ctxChoices(m, 8192, 8192), [8192]); eq(ctxChoices(m, 4096), [4096], "a want at or below the fallback has nothing to fall back to"); - eq(ctxChoices("qwen3.8-27b"), [16384], "no fallback on the 27B"); - eq(ctxChoices("qwen3.6-35b-moe", 131072), [131072]); + eq(ctxChoices("qwen3.8-27b"), [16384, 8192], "the 27B's ladder below its default"); + eq(ctxChoices("qwen3.8-27b", 65536), [65536, 32768, 16384, 8192], "the 27B's whole ladder from its top"); + eq(ctxChoices("qwen3.6-35b-moe", 131072), [131072, 65536, 32768, 16384, 8192], "the MoE's ladder"); + eq(ctxChoices("qwen3.6-35b-moe", 131072, 131072), [131072], "--ctx 131072 is tried alone"); const tried = []; eq(pickCtx(m, { fitsAt: (c) => (tried.push(c), true) }), { ctx: 16384, want: 16384, fits: true, fellBack: false }, "fits at 16K"); eq(tried, [16384], "16K first, nothing else tried when it fits"); @@ -288,10 +290,18 @@ Deno.test("pickCtx: 16K when it fits, 8K fallback when only that fits, an explic eq(pickCtx(m, { fitsAt: () => false }), { ctx: 8192, want: 16384, fits: false, fellBack: false }, "short for both: short for 8K, the least it could start with"); eq(pickCtx(m, { want: 16384, ask: 16384, fitsAt: (c) => c <= 8192 }), { ctx: 16384, want: 16384, fits: false, fellBack: false }, "--ctx 16384 stays 16K (and short)"); eq(pickCtx(m, { want: 8192, ask: 8192, fitsAt: () => true }), { ctx: 8192, want: 8192, fits: true, fellBack: false }, "?ctx=8192 is not a fallback"); - eq(pickCtx("qwen3.8-27b", { fitsAt: () => false }).ctx, 16384, "a model without a fallback keeps its context"); + eq(pickCtx("qwen3.8-27b", { fitsAt: () => false }).ctx, 8192, "the 27B short everywhere: short for its ladder's last step"); + // past the default only where fastAt says so; want / fellBack measured against the default + const T = "qwen3.6-35b-moe", top = 131072; + eq(pickCtx(T, { want: top, fitsAt: () => true, fastAt: () => true }), { ctx: top, want: 32768, fits: true, fellBack: false }, "128K at full speed"); + eq(pickCtx(T, { want: top, fitsAt: () => true, fastAt: (c) => c <= 65536 }), { ctx: 65536, want: 32768, fits: true, fellBack: false }, "64K at full speed, 128K not"); + eq(pickCtx(T, { want: top, fitsAt: () => true, fastAt: () => false }), { ctx: 32768, want: 32768, fits: true, fellBack: false }, "no longer one at full speed: the default"); + eq(pickCtx(T, { want: top, fitsAt: (c) => c <= 16384, fastAt: () => false }), { ctx: 16384, want: 32768, fits: true, fellBack: true }, "short for the default: 16K, fell back"); + eq(pickCtx(T, { want: top, fitsAt: () => true }).ctx, top, "without fastAt (a choice): the longest that fits"); eq(ctxK(16384), "16K"); eq(ctxK(8192), "8K"); eq(ctxShortNote("Qwen3 1.7B", 8192, 16384), "Qwen3 1.7B · 8K context: the room's memory is short for 16K"); eq(needText(m), "5.6 GB (4 GB at 8K)"); - eq(needText("qwen3.8-27b"), `${NEED_GB["qwen3.8-27b"]} GB`); + eq(needText("qwen3.8-27b"), `${NEED_GB["qwen3.8-27b"]} GB (${NEED_MIN_GB["qwen3.8-27b"]} GB at 8K)`); + ok(NEED_MIN_GB["qwen3.8-27b"] < NEED_GB["qwen3.8-27b"] && NEED_MIN_GB["qwen3.6-35b-moe"] < NEED_GB["qwen3.6-35b-moe"]); eq(NEED_MIN_GB[m], 4); }); From 557b8f91f04792e07d67109f621c83e49ac4a330 Mon Sep 17 00:00:00 2001 From: Nehanth Date: Fri, 16 Oct 2026 17:05:55 -0400 Subject: [PATCH 2/3] docs: the context ladder, its trade-off and the GB10 runs at the new default Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MkVLt1BijYxEeNWsgihPdN --- docs/long-context-and-sessions.md | 53 +++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/long-context-and-sessions.md b/docs/long-context-and-sessions.md index efcee4ae..bb0fbf6c 100644 --- a/docs/long-context-and-sessions.md +++ b/docs/long-context-and-sessions.md @@ -53,6 +53,59 @@ each has a switch for A/B timing on real hardware. that binds less than 256 MiB. Needle retrieval and speed at 32K..128K: `tests/needle_ctx.js` and docs/bench-log.md (2026-09-30). +## The 27B and the 35B MoE: a context ladder, the longest at full speed by default (2026-10-16) + +Rooms no longer need `?ctx=` for a long context on the two big hybrids. Each has a ladder of lengths checked correct on it +(`room/models.js` `CTX` `ladder`), and a room opens at the longest step its memory holds (`room/ctxpick.js` +`roomCtxPlan`, called by the picker, Start, the room node behind `pooled host` / `pooled serve` / the OpenClaw plugin, and +`pooled host`'s screen, so they all agree): + +| Model | Ladder | Picker's context row | Old default (`def`) | Why each length is on it | +|---|---|---|---|---| +| Qwen3.8 27B | 64K, 32K, 16K, 8K | 8K, 16K, 32K, 64K | 16K | needle found at 64K (2026-09-30) and 16K / 32K (2026-10-01); 8K is the same engine as 16K / 32K (below). 128K not checked: left out | +| Qwen3.6 35B MoE | 128K, 64K, 32K, 16K, 8K | 8K, 32K, 64K, 128K | 32K | needle found at 32K, 64K, 96K, 128K (2026-09-30); `test_moe.js` MATCHes llama.cpp in a 131072-position engine (below); 8K / 16K are the 32K engine | + +The engine at 32K positions or fewer is one engine: the prefill split length is 256 for any `maxSeq` up to 32K +(`engine/qwen35.js` `faSplit`), and decode's splits follow the filled length (`attn_dec` `decSplits`), so a 8K or 16K room +computes what a 32K room does on any prompt that fits. + +**The trade-off.** A longer window costs memory (f16 K+V: 64 KB per token on the 27B, 20 KB on the MoE, split over the +devices: the 27B needs ~17.0 GB at 16K and ~20 GB at 64K; the MoE ~21.0 GB at 32K and ~22.9 GB at 128K by the planner's +bytes), not speed: decode reads only the filled positions. Measured on the GB10 (Deno `tests/bench_ctx.js`, FILLS=1K,8K, +only `CTX` changed): + +| Model | maxSeq | 1K fill: prefill / plain / spec tok/s | 8K fill: prefill / plain / spec tok/s | tokens | +|---|---|---|---|---| +| MoE | 32768 | 938 / 37.8 / 51.7 | 1159 / 35.6 / 50.6 | | +| MoE | 131072 | 921 / 37.6 / 50.4 | 1155 / 36.3 / 51.1 | identical to 32768 | +| 27B | 16384 | 247 / 10.58 / 19.9 | 277 / 10.23 / 21.3 | | +| 27B | 65536 | 246 / 10.59 / 19.9 | 277 / 10.21 / 21.1 | identical to 16384 | + +`CTX=131072 tests/test_moe.js`: 3 of 3 MATCH llama.cpp, spec = plain (decode 60-61 tok/s; 55-58 at the usual 512 in the +run just before). + +Memory can still cost speed in a room, two ways, so the default goes past `def` only when neither happens +(`roomCtxPlan` `fastAt`, judged on the deal by memory): (1) the longer context leaves no room for the experts, and a device +offloads them (each token then copies experts from RAM or disk); (2) it needs one more device holding layers than at +`def` (a phone, or one more network hop every token). Such a room opens at `def`, and the longer lengths are one click away +in the picker (Room settings changes it later; the model loads again). A room short of memory even for `def` walks the +ladder down (to 8K) and says so; one that started at `def` before still starts there or longer. + +Example rooms (the picker's need per choice: `ctxOptions`, the GPU memory these devices need at that length): + +| Room | Model | 8K | 16K | 32K | 64K | 128K | Opens at | +|---|---|---|---|---|---|---|---| +| Mac 16 + desktop 7 + iPhone 1 (24 GB) | MoE | 20.9 | | 21.4 | 22.1 | 23.5 | 128K, the phone holds nothing | +| Mac 17 + desktop 6 + iPhone 1 (24 GB) | MoE | | | | fits | fits with the phone | 64K (128K would put layers on the phone) | +| Mac 14 + desktop 8 + iPhone 1 (23 GB) | MoE | 21.4 | | 21.8 | 22.5 | needs 0.7 GB more | 64K | +| RTX 12 GB + 48 GB RAM (`pooled host --here`) | MoE | 6.2 | | 6.7 | 7.3 | 8.6 | 32K: experts in RAM at every length, so the default | +| Mac 16 + desktop 8 (24 GB) | 27B | 16.5 | 17.0 | 18.0 | 20.1 | | 64K | +| a 16 GB laptop alone | 27B | short | short | short | short | | short (for 8K) | + +`?ctx=N` and `--ctx N` still ask for an exact length, tried alone. Tests: `tests/unit/ctxpick_test.js` (the picker and +Start agree for these rooms at every choice; the default rule), `cli/test/hostui_test.mjs` (the screen, the room node and +the page agree), `packages/room-node/test/roomnode_test.mjs`. + ## Qwen3.8 Flash in rooms: long context (rooms long-context PR 5, 2026-10-13) Rooms run Flash with QSA and f16 K / V, at up to **64K** (65,536 tokens), the longest context checked against llama.cpp From 6ad62bee97659b7cea2ac124be0555cfdc0ed9f3 Mon Sep 17 00:00:00 2001 From: Nehanth Date: Fri, 16 Oct 2026 17:06:46 -0400 Subject: [PATCH 3/3] docs: example room numbers for the 17 / 6 split Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MkVLt1BijYxEeNWsgihPdN --- docs/long-context-and-sessions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/long-context-and-sessions.md b/docs/long-context-and-sessions.md index bb0fbf6c..c1af6d79 100644 --- a/docs/long-context-and-sessions.md +++ b/docs/long-context-and-sessions.md @@ -96,7 +96,7 @@ Example rooms (the picker's need per choice: `ctxOptions`, the GPU memory these | Room | Model | 8K | 16K | 32K | 64K | 128K | Opens at | |---|---|---|---|---|---|---|---| | Mac 16 + desktop 7 + iPhone 1 (24 GB) | MoE | 20.9 | | 21.4 | 22.1 | 23.5 | 128K, the phone holds nothing | -| Mac 17 + desktop 6 + iPhone 1 (24 GB) | MoE | | | | fits | fits with the phone | 64K (128K would put layers on the phone) | +| Mac 17 + desktop 6 + iPhone 1 (24 GB) | MoE | 21.0 | | 21.5 | 22.1 | 23.3 (with the phone holding layers) | 64K (128K would put layers on the phone) | | Mac 14 + desktop 8 + iPhone 1 (23 GB) | MoE | 21.4 | | 21.8 | 22.5 | needs 0.7 GB more | 64K | | RTX 12 GB + 48 GB RAM (`pooled host --here`) | MoE | 6.2 | | 6.7 | 7.3 | 8.6 | 32K: experts in RAM at every length, so the default | | Mac 16 + desktop 8 (24 GB) | 27B | 16.5 | 17.0 | 18.0 | 20.1 | | 64K |