From 441b45a2a891e2d32d553cf0b620e051aaacb5ca Mon Sep 17 00:00:00 2001 From: Nehanth Date: Fri, 16 Oct 2026 16:53:25 -0400 Subject: [PATCH] Hide the Qwen3.5 122B model from users The 122B stays in room/models.js, the engine and its tests for developers, but is no longer offered or advertised: - CLI: pooled list / host / pull and the join plan list it only with POOLED_DEV=1 (lend.js DEV_MODELS); named without it, host says it is a developer model; pull / rm still take its full key. pooled join still lends to a room that runs it. - Room page: unchanged, only in the ?dev=1 select. A tab now refuses an ai-load for a model its catalogue lacks with a clear ai-error, as the room node already does. - Docs: removed from README, CHANGELOG [3.0.0], docs/releases/3.0.md, cli/README, package READMEs, docs-site pages; docs/models.md keeps one developer-only row. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01MkVLt1BijYxEeNWsgihPdN --- CHANGELOG.md | 6 +++--- README.md | 1 - cli/README.md | 6 +++--- cli/lib/lend.js | 18 ++++++++++++------ cli/lib/lendrun.js | 4 ++-- cli/lib/pullrun.js | 9 +++++---- cli/test/lend_test.mjs | 16 ++++++++++++++++ .../src/content/docs/contributing/models.mdx | 1 - docs-site/src/content/docs/faq.mdx | 2 +- docs-site/src/content/docs/index.mdx | 1 - docs-site/src/content/docs/models.mdx | 9 ++------- docs-site/src/content/docs/openclaw.mdx | 2 +- docs-site/src/content/docs/reference/cli.mdx | 2 +- .../src/content/docs/reference/models.mdx | 13 +++++-------- .../src/content/docs/reference/url-options.mdx | 1 - docs-site/src/content/docs/roadmap.mdx | 2 +- docs-site/src/content/docs/terminal.mdx | 9 ++------- docs/models.md | 4 ++-- docs/openclaw.md | 3 +-- docs/releases/3.0.md | 1 - docs/tech-stack.md | 2 +- packages/openclaw/README.md | 3 +-- packages/room-node/README.md | 4 ++-- room.js | 6 ++++++ 24 files changed, 67 insertions(+), 58 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 06dbcc68..b8359bf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ All notable changes to Pooled (called SwarmLLM before September 2026). Format fo ## [3.0.0] - 2026-10-16 -**Run a 125B model from your SSD, in a browser tab, and pool devices to make it faster.** Pooled 3.0 runs Qwen3.8 Flash (**experimental**), a 125B mixture-of-experts model with about 6B active per token. It is far bigger than most GPUs' memory, so each device keeps its share of the experts on its SSD (browser storage for a tab, the model files for `pooled join`), caches recently used experts on its GPU (routing repeats: about 17% of a layer's experts serve about 77% of lookups), and reads the rest from disk as tokens need them. Disk is the slow part; every device that joins holds part of the model in its memory, so less comes from disk and it gets faster: a Mac Studio alone in Chrome decodes about 8 tok/s, and with an RTX 5070 PC joined about 10-12 tok/s, depending on the network. Flash rooms run at up to 64K context (128K is in progress). 3.0 also brings the system-RAM and SSD expert tiers in the terminal, the Qwen3.5 122B MoE, `pooled serve`, channel-bound join auth and the default TURN relay. Release notes and the demo video: [docs/releases/3.0.md](docs/releases/3.0.md). +**Run a 125B model from your SSD, in a browser tab, and pool devices to make it faster.** Pooled 3.0 runs Qwen3.8 Flash (**experimental**), a 125B mixture-of-experts model with about 6B active per token. It is far bigger than most GPUs' memory, so each device keeps its share of the experts on its SSD (browser storage for a tab, the model files for `pooled join`), caches recently used experts on its GPU (routing repeats: about 17% of a layer's experts serve about 77% of lookups), and reads the rest from disk as tokens need them. Disk is the slow part; every device that joins holds part of the model in its memory, so less comes from disk and it gets faster: a Mac Studio alone in Chrome decodes about 8 tok/s, and with an RTX 5070 PC joined about 10-12 tok/s, depending on the network. Flash rooms run at up to 64K context (128K is in progress). 3.0 also brings the system-RAM and SSD expert tiers in the terminal, `pooled serve`, channel-bound join auth and the default TURN relay. Release notes and the demo video: [docs/releases/3.0.md](docs/releases/3.0.md). Not in 3.0: the CLI on npm (`@pooled/cli` 0.4.1) doesn't have Flash's newest code yet; to lend or host Flash from the terminal, run the CLI from the repo ([cli/README.md](cli/README.md#qwen38-flash-experimental-run-the-cli-from-the-repo)). Flash MTP speculative decoding (#352) is not part of 3.0. @@ -33,9 +33,9 @@ Not in 3.0: the CLI on npm (`@pooled/cli` 0.4.1) doesn't have Flash's newest cod - **An explicit 128 GB pledge cap for Qwen3.8-Flash-Next only** (#356, #369; `room/pledge.js` `PLEDGE_MAX_GB = 128`, `maxPledgeGB`): one big unified-memory machine lending from the terminal (`pooled host` / `pooled join`) can hold Flash alone; a browser tab still lends at most 64 GB. Every other model still counts at most 64 GB per device, and the default memory rule is unchanged. `pooled join --gb max` may still say up to 128 GB, because a joiner doesn't know the room's model yet; the host counts 64 of it for non-Flash models. - **The speed panel** (#353, #358): live tok/s, device cards, per-hop timing from room nodes and the expert-cache hit rate. - **Expert offload's per-layer cut in Chrome: 2.9 ms → 0.5 ms** (#351): a map kick (on by default, Chrome only), a pipelined cut (`cutPipe`, opt-in) and speculative windows. -- **`--disk`: MoE experts read from the SSD when the GPU and RAM are short (roadmap 32, steps 3-5; the 35B, the 122B and Flash, #368).** `pooled join --disk` and `pooled host --disk` (with `--here` too) let a computer with a discrete GPU and `--ram` leave some layers' routed experts in the model's files and read them when a token picks them, through a RAM cache of per-expert upload buffers (at least 2 GB of the `--ram`; segmented LRU, an expert is cached on its second miss). The room uses it only when parking experts in RAM can't hold the model, and a room that fits without it deals exactly as before. Outputs stay bit-exact (the 35B with every layer on disk gives main's hashes). +- **`--disk`: MoE experts read from the SSD when the GPU and RAM are short (roadmap 32, steps 3-5; the 35B and Flash, #368).** `pooled join --disk` and `pooled host --disk` (with `--here` too) let a computer with a discrete GPU and `--ram` leave some layers' routed experts in the model's files and read them when a token picks them, through a RAM cache of per-expert upload buffers (at least 2 GB of the `--ram`; segmented LRU, an expert is cached on its second miss). The room uses it only when parking experts in RAM can't hold the model, and a room that fits without it deals exactly as before. Outputs stay bit-exact (the 35B with every layer on disk gives main's hashes). - **The CLI uses the TURN relay too.** `pooled host`, `pooled join`, `pooled chat` and `pooled serve` (and the OpenClaw plugin's room node) used to try direct links only (public STUN), so two devices behind strict NATs or CGNAT could not reach each other from the terminal, though the room page could through pooled.run's relay. They now fetch the same short-lived relay credentials (`cli/lib/relay.js`) and use the relay only when no direct path works, refreshing the credentials at 3/4 of their lifetime. `/api/turn` lets a request with no `Origin` through when it names itself in `X-Pooled-Client` (a browser on another site always sends `Origin`, so other pages still can't mint credentials; the per-IP limit applies). `POOLED_RELAY=0` / `1` turns the relay off / forces it, `POOLED_TURN` (+ `POOLED_TURN_USER`, `POOLED_TURN_CRED`) uses your own, `POOLED_TURN_API` another deployment's. Tested with every link forced through Cloudflare's relay: `pooled host` and `pooled chat` on a GB10 answered over relayed links only. -- **`@pooled/cli` 0.4.0 and `@pooled/openclaw` 0.3.0: MoE experts in system RAM, and the Qwen3.5 122B MoE.** A `pooled host` / `pooled join` device with a discrete GPU keeps some layers' routed experts in system RAM when the room's pledges can't hold a MoE model, and copies the ones each token uses into a cache on its GPU (#319, #320). `pooled host qwen3.6-35b-moe --here` now runs on a 12 GB GPU (about 38-45 tok/s on an RTX 5070), and a Mac Studio lending 25 GB with an RTX 5070 PC keeping 42 GB of experts in RAM runs the 122B at about 17 tok/s. `--ram N` sets how much RAM (default: the total less 16 GB, at most what is free less 4 GB, and 64 GB; `--ram 0` turns it off); unified memory and browsers never offload. While a device offloads, the room decodes without speculation (faster there; `POOLED_OFFLOAD_SPEC=1` turns it back on), and that device keeps GPU and RAM headroom past its cache (#323). Qwen3.5-122B-A10B (a two-file GGUF, 72.5 GB) is in `pooled pull` and `pooled host`'s list, with what it needs, and in the room page's `?dev=1` list (#316). On Windows the room node runs V8 without Maglev, which crashed Dawn (#317). The plugin bundles the same room node, so its rooms deal offload to `pooled join` devices (the gateway itself holds its layers whole), and its README says Windows is tested (#322). +- **`@pooled/cli` 0.4.0 and `@pooled/openclaw` 0.3.0: MoE experts in system RAM.** A `pooled host` / `pooled join` device with a discrete GPU keeps some layers' routed experts in system RAM when the room's pledges can't hold a MoE model, and copies the ones each token uses into a cache on its GPU (#319, #320). `pooled host qwen3.6-35b-moe --here` now runs on a 12 GB GPU (about 38-45 tok/s on an RTX 5070). `--ram N` sets how much RAM (default: the total less 16 GB, at most what is free less 4 GB, and 64 GB; `--ram 0` turns it off); unified memory and browsers never offload. While a device offloads, the room decodes without speculation (faster there; `POOLED_OFFLOAD_SPEC=1` turns it back on), and that device keeps GPU and RAM headroom past its cache (#323). On Windows the room node runs V8 without Maglev, which crashed Dawn (#317). The plugin bundles the same room node, so its rooms deal offload to `pooled join` devices (the gateway itself holds its layers whole), and its README says Windows is tested (#322). ### Changed - **Flash prompt reading ~8x on the GB10** (#377): wide kernels (tiled GEMMs, expert-grouped MoE, GQA-shared attention) take the whole resident model from ~40 to ~330 tok/s; a 32K prompt in ~100 s instead of ~14 min. Rooms batch Flash prompts across the pipeline and in solo rooms (#381): ~190-200 tok/s on a two-node GB10 room at 2K, was ~21. diff --git a/README.md b/README.md index 2a987fa3..5bab3590 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,6 @@ Models in the room today: | Qwen 3.8 27B | Q4_0, needs ~17 GB across the room | hybrid Gated DeltaNet + attention, built-in draft layer for speculative decoding, 16K context (up to 64K) | | Qwen 3.6 35B MoE | Q4_0, needs ~22.5 GB across the room | 256 experts, 8 active per token, so it decodes several times faster than the 27B; 32K context (up to 128K) | | Qwen3 1.7B | Q8_0, needs ~5.6 GB across the room for 16K context, ~4 GB for 8K | small and quick, for rooms of phones and light laptops; 16K context, 8K when the room is short of memory | -| Qwen3.5 122B MoE | Q4_0, a 72.5 GB download, needs ~73 GB across the room | 256 experts, 8 active per token; in `pooled pull` / `pooled host` (CLI 0.4.0+), on the room page only with `?dev=1`. A Mac Studio plus an RTX 5070 PC keeping its experts in RAM runs it at ~17 tok/s | | Qwen3.8 Flash (**experimental**) | Q4_0, 125B (6B active), ~99 GB on disk | 512 experts, 10 active per token; keeps most of itself on disk, so it needs far less GPU memory than its size; 64K context. See below | Every device downloads only its own layers, from Hugging Face or from another device in the room that already has them, and keeps them cached for next time. If a device leaves, the room says so and the host deals its layers out again with one click. Speculative decoding, batched prefill and every kernel trick are checked by golden tests, so the speculative stream is the same as plain decoding. diff --git a/cli/README.md b/cli/README.md index 0fe6b1ff..1dd80354 100644 --- a/cli/README.md +++ b/cli/README.md @@ -303,7 +303,7 @@ Not downloaded (pooled pull ): … ``` -The list includes the Qwen3.5 122B MoE (`qwen3.5-122b-moe`, two files, 72.5 GB): `pooled host` says it needs about 73 GB across the room, for a room of big machines or one with a PC that keeps its experts in RAM (below). The room page lists it only with `?dev=1`. Qwen3.8-Flash-Next (`qwen3.8-flash`, three files, 100.6 GB, of which 28.8 GB is an n-gram table that stays on the disk of the device holding layer 1) is **experimental** (the CLI calls it a preview): `pooled host` and `list` show it only with `POOLED_DEV=1` (named without it, they say so). `pooled join` needs no flag to lend to a room that runs it, and `pooled pull qwen3.8-flash` / `pooled rm qwen3.8-flash` take its full name without `POOLED_DEV=1`, so a device can have its files before it joins. It needs about 72 GB (GiB, the unit pledges use) across the room at its longest proven context, 64K (about 67 GB at 2K: a room short for 64K opens at the longest length it holds); a `pooled` device with a discrete GPU parks its experts in RAM (and reads them from the files with the disk tier) as for the other MoE models. One GB10 holds it alone with `POOLED_DEV=1 pooled host qwen3.8-flash --here`. +Qwen3.8-Flash-Next (`qwen3.8-flash`, three files, 100.6 GB, of which 28.8 GB is an n-gram table that stays on the disk of the device holding layer 1) is **experimental** (the CLI calls it a preview): `pooled host` and `list` show it only with `POOLED_DEV=1` (named without it, they say so). `pooled join` needs no flag to lend to a room that runs it, and `pooled pull qwen3.8-flash` / `pooled rm qwen3.8-flash` take its full name without `POOLED_DEV=1`, so a device can have its files before it joins. It needs about 72 GB (GiB, the unit pledges use) across the room at its longest proven context, 64K (about 67 GB at 2K: a room short for 64K opens at the longest length it holds); a `pooled` device with a discrete GPU parks its experts in RAM (and reads them from the files with the disk tier) as for the other MoE models. One GB10 holds it alone with `POOLED_DEV=1 pooled host qwen3.8-flash --here`. A download is written to `.part` and resumes where it stopped (Ctrl-C, a lost network: run `pooled pull` again, it asks for the rest with a Range request); the file is checked against its size and SHA-256 before it is renamed and used. `pooled download` is the same command, `pooled ls` the same as `pooled list`, and `pooled rm ` deletes one. A part of a name works: `pooled pull 35b`. Without a terminal the progress is a plain line every few seconds. @@ -379,7 +379,7 @@ Lending 9.5 GB of GPU memory (11.4 GB free, 1.5 GB kept free) and 43 GB of RAM f - **GPU memory** (`--gb`): the memory rule below (a discrete GPU's free memory less 1.5 GB; unified memory the total less max(8 GB, 35%)). A join may lend past 64 GB by itself (a big Mac or GB10 in a Qwen3.8-Flash-Next room); the host counts at most its model's cap of it (64 GB for every other model). - **RAM for experts** (`--ram`, discrete GPUs, MoE rooms only): the total less max(16 GB, 25%), never more than is free now less 4 GB, at most 64. A 16 GB laptop lends none; a 64 GB PC with 47 GB free lends 43. - **Disk tier** (`--disk`, on by itself; `--no-disk` turns it off): offered wherever `--disk` works (a discrete GPU, 2 GB or more of RAM for experts). The host reads experts from disk only for a model whose files are all on this computer, and only when parking them in RAM can't hold the room, so it changes nothing otherwise. A model downloaded after joining is announced to the host at once. -- **The model's files**: on this computer, it loads from disk. Not there, it downloads the model first when it is 40 GB or smaller and the disk keeps 20 GB free after it; otherwise (Qwen3.8-Flash-Next's 100.6 GB, the 122B's 72.5 GB, a full disk) it streams only this device's layers from Hugging Face, with its experts in RAM and no disk tier, and says why. `--pull` downloads any size first; `pooled pull ` beforehand does it without holding up the room. +- **The model's files**: on this computer, it loads from disk. Not there, it downloads the model first when it is 40 GB or smaller and the disk keeps 20 GB free after it; otherwise (Qwen3.8-Flash-Next's 100.6 GB, a full disk) it streams only this device's layers from Hugging Face, with its experts in RAM and no disk tier, and says why. `--pull` downloads any size first; `pooled pull ` beforehand does it without holding up the room. - `pooled join --dry-run` prints all of that for this computer (and, per model, whether it would load, download or stream it), touching neither the GPU nor the network, then exits. `pooled join` alone, in a terminal, asks for the code or the invite link, shows the host and its model ("downloaded here: loads from disk" when you pulled it) and asks how much to lend. When the host's model is not in `~/.pooled/models` yet (and the rule above says to download it), it downloads the whole file there first, with the same progress bar as `pooled pull` ("Qwen3 1.7B isn't downloaded yet (1.7 GB). Downloading it into ~/.pooled/models so this and later joins load from disk."), then loads from disk; a deal that comes meanwhile waits for it. `--no-pull` streams only this device's layers from Hugging Face instead, and the status line then says what it is doing: `downloading 312 of 900 MB · 18 MB/s · from Hugging Face`, then `loading onto the GPU`. @@ -404,7 +404,7 @@ room HJQ-N44 · online · 2 devices · layers 14-27 of qwen3-1.7b · 56.4 tok/s - **Getting in** (the room page's gate, [docs/protocol.md](../docs/protocol.md) "Joining a room"; the room page's side is PR #268). Room codes are six characters (`4TK-G9P`; four-character codes of older rooms still work), and a code only finds a room. A device that has the room's **invite link** (its `#k=` key; quote it in the shell) is let in at once; one with the code alone waits in the host's lobby, "waiting for the host to let you in", until the host allows it. The host gives each device it lets in a pass, so a device that loses its link and knocks again is let back in without asking. `pooled join`, `pooled chat` and `pooled serve` all take the link. A host from before the gate lets everyone in, as before. - **As a host**, `pooled host` prints the invite link and asks you about each device that comes with the code alone: `a` lets the oldest request in, `d` turns it away. Without a terminal nobody is asked, so such a device waits until you give it the link; `--allow-all` lets in anyone with the code (and room pages from before the gate, which can't wait in a lobby), `--deny-unknown` turns them away. Enter deals the layers over the devices in the room once their pledges hold the model (and again, re-deals after more join); `--wait N` deals as soon as N devices are in and fit, and deals again by itself when a device stayed away past the minute's grace (the room went on without it) and N are back. - **How much it lends.** WebGPU does not say how much memory a GPU has, so `pooled` asks the OS: `nvidia-smi` for NVIDIA cards, sysfs for AMD on Linux, the system's memory on Apple silicon and on GPUs that share it (GB10). A discrete GPU lends its free memory less 1.5 GB; unified memory lends the total less max(8 GB, 35%) (on Linux, never more than is free right now, less 2 GB); at most 64 GB per device. For Qwen3.8-Flash-Next only, `--gb N` and `--gb max` (and `--here`, when the model needs it) go up to 128 GB, so one big unified-memory machine can hold it alone; a `pooled join` that says more than 64 counts as 64 in a room of any other model. On a discrete GPU it then allocates that much for a moment to make sure it is there (`--no-check` skips this; unified memory skips it, since the OS's numbers are the memory itself and touching 64 GB of it takes about 20 s). `--gb N` sets it, `--gb max` keeps only a small margin. When the memory can't be read it lends half of the GPU's largest buffer and says so. -- **Expert offload (MoE models, discrete GPUs).** A MoE layer is mostly its 256 routed experts, of which a token uses 8. When the room's pledges can't hold a MoE model (the 35B or the 122B), a `pooled` device with a discrete GPU holds layers past its pledge anyway: it keeps those layers' experts in system RAM and copies the ones each token picks into a cache on the GPU (slower than layers held whole: one GPU round trip per offloaded layer per token, plus the copies on a cache miss). This is what lets a 12 GB GPU run the 35B alone with `pooled host qwen3.6-35b-moe --here` (it lends the memory rule's GB; the cache gets what the layers don't use): about 38-45 tok/s on an RTX 5070, with 17.5 GiB of experts in RAM. A Mac Studio lending 25 GB and an RTX 5070 PC keeping 42 GB of experts in RAM run the 122B at about 17 tok/s. How much RAM it may use is `--ram N`, like `--gb` for the GPU: by default the total less max(16 GB, 25%), never more than is free at start less 4 GB, at most 64 GB; `--ram 0` turns it off. Unified memory (Apple silicon, GB10) never offloads: its pledge is that same memory. The host deals it (room/plan.js, the room page does the same): a room the pledges hold deals exactly as before; otherwise every device that doesn't offload holds all its pledge holds, and the devices that offload take the rest, each within its pledge (the layers' other weights plus the cache: one whole layer's experts for prompts and at least 16 cached experts per offloaded layer) and within its `--ram`. It offloads the fewest layers that still leave each a 64-expert cache (a quarter of its experts), else as many as its RAM takes. A device that offloads also keeps room past its layers and its cache: about 2 GB of its pledge for the engine's own buffers and the prompt checkpoints, and about 1.9 GB of its `--ram` for the process past the parked experts (the 122B on a 12 GB GPU: a 5.2 GB cache instead of 7.2). Without that, a long prompt pushed the GPU past its memory and Windows moved the overflow into system RAM (6 GB on the RTX 5070 PC). While a device offloads, the room decodes without speculation (draft and lookup tokens): a verify pass makes it copy in several times the experts a plain token does, so plain decoding is faster there (the 122B: 17-18 tok/s plain, 9-13 with speculation). `POOLED_OFFLOAD_SPEC=1` on the host (`?offspec=1` on a room page host) turns it back on. The header shows `RAM up to N GB ...`, the host's log which layers each device offloads. Browsers don't offload. +- **Expert offload (MoE models, discrete GPUs).** A MoE layer is mostly its 256 routed experts, of which a token uses 8. When the room's pledges can't hold a MoE model (the 35B), a `pooled` device with a discrete GPU holds layers past its pledge anyway: it keeps those layers' experts in system RAM and copies the ones each token picks into a cache on the GPU (slower than layers held whole: one GPU round trip per offloaded layer per token, plus the copies on a cache miss). This is what lets a 12 GB GPU run the 35B alone with `pooled host qwen3.6-35b-moe --here` (it lends the memory rule's GB; the cache gets what the layers don't use): about 38-45 tok/s on an RTX 5070, with 17.5 GiB of experts in RAM. How much RAM it may use is `--ram N`, like `--gb` for the GPU: by default the total less max(16 GB, 25%), never more than is free at start less 4 GB, at most 64 GB; `--ram 0` turns it off. Unified memory (Apple silicon, GB10) never offloads: its pledge is that same memory. The host deals it (room/plan.js, the room page does the same): a room the pledges hold deals exactly as before; otherwise every device that doesn't offload holds all its pledge holds, and the devices that offload take the rest, each within its pledge (the layers' other weights plus the cache: one whole layer's experts for prompts and at least 16 cached experts per offloaded layer) and within its `--ram`. It offloads the fewest layers that still leave each a 64-expert cache (a quarter of its experts), else as many as its RAM takes. A device that offloads also keeps room past its layers and its cache: about 2 GB of its pledge for the engine's own buffers and the prompt checkpoints, and about 1.9 GB of its `--ram` for the process past the parked experts. Without that, a long prompt pushed the GPU past its memory and Windows moved the overflow into system RAM (6 GB on the RTX 5070 PC). While a device offloads, the room decodes without speculation (draft and lookup tokens): a verify pass makes it copy in several times the experts a plain token does, so plain decoding is faster there. `POOLED_OFFLOAD_SPEC=1` on the host (`?offspec=1` on a room page host) turns it back on. The header shows `RAM up to N GB ...`, the host's log which layers each device offloads. Browsers don't offload. - **Experts read from disk (`--disk`).** When even the RAM can't hold the experts, a device started with `--disk` (on `pooled join` or `pooled host`, `--here` too) leaves the experts of its last offloaded layers in the model's files and reads them from the SSD when a token picks them (worker threads, several reads at once), through a RAM cache of per-expert upload buffers: an expert is cached the second time it misses, so a prompt's one-off experts don't push out the ones decode keeps using. The host uses it only as a last resort: a room that fits with experts parked in RAM deals exactly as without it, each device parks all its `--ram` holds first, and at least 2 GB of the `--ram` goes to the cache. It needs the model's files on the device (pulled; not `--no-pull`) and a discrete GPU; unified memory doesn't do it yet. `pooled join` offers it by itself (`--no-disk` turns that off); `pooled host` only with `--disk`. Slower than parked experts, and outputs are the same bits. - **The status line** shows the room, what is happening (waiting for the host to let you in, waiting for the host to deal layers, loading, online, answering, a device left, rejoining), the devices in the room, the layers this computer holds, the room's speed on its last answer and how many passes ran here. Without a terminal (or with `--json-log`) it prints a line when that changes, and once a minute. - **When the link drops.** A lost signaling server does not stop the room (the links are direct); `pooled` reconnects to it with backoff (2, 4, 8, 16, 30 s), as the room page does, and tries the next server in a `--signal` list when one is down. When the link to the host drops it knocks for a minute (with its pass) and the host puts it back in its slot, with its layers still loaded. If the host does not come back, it joins the room again from scratch for `--wait` minutes (10 by default), then stops. It goes back only to a host of the same name: a room code can be reused by a new room, and `pooled join` never joins a room you did not name (it leaves one with another host and says so). Without `--name`, the device's name is `node-` and 3 letters made from the hostname, the same on every run, so a `pooled join` started again after a crash takes back its slot. diff --git a/cli/lib/lend.js b/cli/lib/lend.js index 72fe1781..c7744a25 100644 --- a/cli/lib/lend.js +++ b/cli/lib/lend.js @@ -334,17 +334,23 @@ export function ctxNote(rn, model, ask) { return `the ${name}'s context is ${got}; using that`; } // The models pooled host / pull / list offer. A preview model's kind (Qwen3.8-Flash-Next, qwen4exp: its own engine, -// context <= 2048, no MTP in rooms yet) is listed and accepted only with POOLED_DEV=1, as the room page lists the 122B -// only with ?dev=1. A room node still loads one a host deals it (pooled join). +// context <= 2048, no MTP in rooms yet) is listed and accepted only with POOLED_DEV=1, as the room page lists the +// models outside its picker only with ?dev=1. DEV_MODELS: models kept for developers only, hidden the same way +// (Qwen3.5 122B MoE: still in room/models.js and the engine's tests, not offered to users). pooled pull / rm take +// either named in full without the flag (pullrun.js), and a room node still loads one a host deals it (pooled join). export const DEV_KINDS = ["qwen4exp"]; +export const DEV_MODELS = ["qwen3.5-122b-moe"]; export const devOn = (env = process.env) => env.POOLED_DEV === "1"; -export const hostable = (models, dev = devOn()) => Object.keys(models).filter((k) => models[k].kind === "gguf" || models[k].kind === "qwen35" || (dev && DEV_KINDS.includes(models[k].kind))); -// a preview model named without POOLED_DEV=1: what to say ("" for any other name) +export const devModel = (models, k) => DEV_KINDS.includes(models[k]?.kind) || DEV_MODELS.includes(k); +export const hostable = (models, dev = devOn()) => Object.keys(models).filter((k) => + (models[k].kind === "gguf" || models[k].kind === "qwen35" || DEV_KINDS.includes(models[k].kind)) && (dev || !devModel(models, k))); +// a preview or developer model named without POOLED_DEV=1: what to say ("" for any other name) export function devHint(models, given, dev = devOn()) { if (dev || !models) return ""; const t = String(given || "").trim().toLowerCase(); - const hit = Object.keys(models).filter((k) => DEV_KINDS.includes(models[k].kind) && (k === t || (t.length > 2 && k.includes(t)))); - return hit.length === 1 ? `${hit[0]} is a preview: set POOLED_DEV=1 to use it` : ""; + const hit = Object.keys(models).filter((k) => devModel(models, k) && (k === t || (t.length > 2 && k.includes(t)))); + if (hit.length !== 1) return ""; + return DEV_MODELS.includes(hit[0]) ? `${hit[0]} is a developer model: set POOLED_DEV=1 to use it` : `${hit[0]} is a preview: set POOLED_DEV=1 to use it`; } // ---------------- the memory rule ---------------- diff --git a/cli/lib/lendrun.js b/cli/lib/lendrun.js index 108801b9..e8844ecb 100644 --- a/cli/lib/lendrun.js +++ b/cli/lib/lendrun.js @@ -5,7 +5,7 @@ import { execFileSync } from "node:child_process"; import { readFileSync, statfsSync } from "node:fs"; import os from "node:os"; import { parseLendArgs, parseGb, detectMemory, memoryRule, ramRule, diskRule, afterCheck, formatStatus, tpsFromStats, explainError, versionFromBye, - hostable, devHint, ctxNote, autoRedeal, fmtGb, fmtCode, passCounter, deviceName, UsageError, HELP_JOIN, HELP_HOST, needsRoom, loadText, + hostable, devHint, devOn, DEV_MODELS, ctxNote, autoRedeal, fmtGb, fmtCode, passCounter, deviceName, UsageError, HELP_JOIN, HELP_HOST, needsRoom, loadText, pullPlan, autoLine, LEND_MAX_GB } from "./lend.js"; import { dawnLoader, quietLoader, driverLog } from "./dawn.js"; import { cleanText } from "./common.js"; @@ -147,7 +147,7 @@ async function runDry(opts) { if (opts.disk && !disk.on) w(` (${disk.why})`); w(""); w("In a room running:"); - const keys = Object.keys(rn.MODELS).filter((k) => rn.FILES?.[k] && (hostable(rn.MODELS, true).includes(k))); + const keys = Object.keys(rn.MODELS).filter((k) => rn.FILES?.[k] && hostable(rn.MODELS, true).includes(k) && (devOn() || !DEV_MODELS.includes(k))); for (const k of keys) { const m = modelPlan(rn, k, opts); const p = m.plan; diff --git a/cli/lib/pullrun.js b/cli/lib/pullrun.js index 52fb7d43..d043d22e 100644 --- a/cli/lib/pullrun.js +++ b/cli/lib/pullrun.js @@ -7,7 +7,7 @@ import { style, label, progressRow, padStart, clip, gb, I } from "./style.js"; import { liveRegion } from "./tui.js"; import { colorOn } from "./tui.js"; import { argsError } from "./cli.js"; -import { hostable, devHint, DEV_KINDS, fmtGb } from "./lend.js"; +import { hostable, devHint, devModel, fmtGb } from "./lend.js"; import { statfsSync } from "node:fs"; export const HELP_PULL = `Usage @@ -112,10 +112,11 @@ export async function pullMain(cmd0, argv, { loadRoomNode }) { const pick = () => { if (!r.positionals.length) { err(`pooled ${cmd0} needs a model: pooled ${cmd0} ${keys.includes("qwen3-1.7b") ? "qwen3-1.7b" : keys[0]} (pooled list shows them all).`); return null; } if (r.positionals.length > 1) { err(`pooled ${cmd0} takes one model, not ${r.positionals.length}: ${r.positionals.slice(0, 3).join(" ")}`); return null; } - // a preview model (Qwen3.8-Flash-Next) named in full: pooled pull / rm take it without POOLED_DEV=1, so a device can - // have it on disk before it joins a room that runs it (pooled join reads its experts from those files, --disk) + // a preview or developer model (Qwen3.8-Flash-Next, the Qwen3.5 122B: lend.js devModel) named in full: pooled pull / rm + // take it without POOLED_DEV=1, so a device can have it on disk before it joins a room that runs it (pooled join reads + // its experts from those files, --disk), and can free the disk an older pooled filled with it const exact = String(r.positionals[0]).trim().toLowerCase(); - if (cmd !== "list" && rn.MODELS[exact]?.gguf && DEV_KINDS.includes(rn.MODELS[exact].kind)) return exact; + if (cmd !== "list" && rn.MODELS[exact]?.gguf && devModel(rn.MODELS, exact)) return exact; const m = resolveModel(r.positionals[0], keys); if (m.error) { err(`pooled ${cmd0}: ${devHint(rn.MODELS, r.positionals[0]) || `${m.error}; one of: ${m.choices.join(", ")}`}`); return null; } return m.key; diff --git a/cli/test/lend_test.mjs b/cli/test/lend_test.mjs index 5d75e4b8..f673536a 100644 --- a/cli/test/lend_test.mjs +++ b/cli/test/lend_test.mjs @@ -82,6 +82,22 @@ test("Qwen3.8-Flash-Next is a preview: listed and accepted only with POOLED_DEV= } finally { if (prev == null) delete process.env.POOLED_DEV; else process.env.POOLED_DEV = prev; } }); +test("the Qwen3.5 122B MoE is a developer model: hidden without POOLED_DEV=1, and named it says so", () => { + const M = { ...MODELS, "qwen3.5-122b-moe": { kind: "qwen35" } }; + assert.deepEqual(hostable(M, false), ["qwen3-1.7b", "qwen3.6-35b-moe"]); + assert.deepEqual(hostable(M, true), ["qwen3-1.7b", "qwen3.6-35b-moe", "qwen3.5-122b-moe"]); + assert.equal(devHint(M, "qwen3.5-122b-moe", false), "qwen3.5-122b-moe is a developer model: set POOLED_DEV=1 to use it"); + assert.equal(devHint(M, "122b", false), "qwen3.5-122b-moe is a developer model: set POOLED_DEV=1 to use it"); + assert.equal(devHint(M, "qwen3.5-122b-moe", true), ""); + const prev = process.env.POOLED_DEV; + try { + delete process.env.POOLED_DEV; + assert.throws(() => parseLendArgs("host", ["122b"], { models: M }), /qwen3\.5-122b-moe is a developer model: set POOLED_DEV=1/); + process.env.POOLED_DEV = "1"; + assert.equal(parseLendArgs("host", ["122b"], { models: M }).model, "qwen3.5-122b-moe"); + } finally { if (prev == null) delete process.env.POOLED_DEV; else process.env.POOLED_DEV = prev; } +}); + test("--gb: numbers, max, and the per-device cap", () => { assert.equal(parseGb(undefined), null); assert.deepEqual(parseGb("MAX"), { max: true }); diff --git a/docs-site/src/content/docs/contributing/models.mdx b/docs-site/src/content/docs/contributing/models.mdx index 90ae5bec..f03fdfd8 100644 --- a/docs-site/src/content/docs/contributing/models.mdx +++ b/docs-site/src/content/docs/contributing/models.mdx @@ -28,7 +28,6 @@ Pooled ships no weights: browsers fetch tensors from Hugging Face with range req |---|---|---|---| | Qwen 3.8 27B | GGUF Q4_0, about 15 GB, with the `nextn` draft layer | `Qwen35Engine` | 64 layers: 48 Gated DeltaNet and 16 attention. Speculative decoding with the built-in draft layer. | | Qwen 3.6 35B MoE (A3B) | GGUF Q4_0 (bartowski) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active per token. Router, experts and shared expert run on the GPU. | -| Qwen3.5 122B-A10B MoE | GGUF | `Qwen35Engine` (`qwen35moe`) | Runs in the engine and its tests. Not in the room's picker. | | Qwen3 4B, 1.7B, 0.6B | GGUF Q4_0 or Q8_0 | `DenseEngine` | 0.6B is the golden-test model. | | SmolLM2 135M | safetensors f32 | `DenseEngine` | The smallest demo. Quantized to Q8 at load. | diff --git a/docs-site/src/content/docs/faq.mdx b/docs-site/src/content/docs/faq.mdx index a5337d52..899c4b53 100644 --- a/docs-site/src/content/docs/faq.mdx +++ b/docs-site/src/content/docs/faq.mdx @@ -65,7 +65,7 @@ Yes, on every device that holds layers. Pooled asks the browser to keep it on. I ### Which model should I pick? -The biggest one the room has memory for: Qwen3 1.7B at 4 GB (8K context; 5.6 GB for 16K), Qwen3.8 27B at 17 GB, Qwen3.6 35B MoE at 22.5 GB. The picker preselects it. Two bigger MoE models are newer: Qwen3.5 122B MoE at about 73 GB (a preview, behind `?dev=1`, and in the terminal), and Qwen3.8-Flash-Next (125B, experimental), a picker row of its own that needs about 8 GB of GPU memory on one tab that keeps its experts on disk (it asks first). The MoE is also the fastest of the big two. See [Pick a model](/docs/models). +The biggest one the room has memory for: Qwen3 1.7B at 4 GB (8K context; 5.6 GB for 16K), Qwen3.8 27B at 17 GB, Qwen3.6 35B MoE at 22.5 GB. The picker preselects it. A bigger MoE model is newer: Qwen3.8-Flash-Next (125B, experimental), a picker row of its own that needs about 8 GB of GPU memory on one tab that keeps its experts on disk (it asks first). The MoE is also the fastest of the big two. See [Pick a model](/docs/models). ### Why is it slower than running the model natively? diff --git a/docs-site/src/content/docs/index.mdx b/docs-site/src/content/docs/index.mdx index 41fc31ca..b49bb3a4 100644 --- a/docs-site/src/content/docs/index.mdx +++ b/docs-site/src/content/docs/index.mdx @@ -33,7 +33,6 @@ One laptop is enough for the smallest model. More devices let you run bigger one | Qwen3 1.7B | 5.6 GB at a 16K context, 4 GB at 8K | | Qwen3.8 27B | 17 GB | | Qwen3.6 35B MoE | 22.5 GB | -| Qwen3.5 122B MoE (preview, `?dev=1` or the terminal) | about 73 GB | | Qwen3.8-Flash-Next (experimental) | about 72 GB on GPUs, or about 8 GB for one tab that keeps its experts on disk (up to 92.4 GB of disk per device) | ## What you can do with a room diff --git a/docs-site/src/content/docs/models.mdx b/docs-site/src/content/docs/models.mdx index 24406428..36df5ebc 100644 --- a/docs-site/src/content/docs/models.mdx +++ b/docs-site/src/content/docs/models.mdx @@ -9,10 +9,8 @@ sidebar: {/* Sources (for the fact-checker): - room/models.js: PICKER, NEED_GB (5.6 / 17.0 / 22.5), NEED_MIN_GB (1.7B 4.0 at its 8K fallback), CTX.fallback + pickCtx() (16K, else 8K when the pledges are short for 16K), FILE_GB (1.83 / 16.06 / 20.84), CTX (1.7B 16384/16384, fallback 8192, 27B 16384/32768, MoE 32768/65536), maxSeqFor() (?ctx= rounded to 256, at least 2048) -- 0.4.0 (origin/main ed31899): CTX 27B max 65536, MoE max 131072; the 122B (#316): NEED_GB 72 (the CLI picker's "needs 73 GB" - with its context), FILE_GB 72.52, in hostable() for pooled host / pull, ?dev=1 only on the room page; expert offload - (#319, #320, #323): 35B on the RTX 5070 alone about 38-45 tok/s (37.5-44.9 measured on the published 0.4.0, ~40 on long generations), 17.5 GiB parked; - 122B Mac Studio --gb 25 + RTX 5070 PC (layers 15-47 offloaded, 41.8 GiB parked) ~17 tok/s (#323, maintainer's run) +- 0.4.0 (origin/main ed31899): CTX 27B max 65536, MoE max 131072; expert offload + (#319, #320, #323): 35B on the RTX 5070 alone about 38-45 tok/s (37.5-44.9 measured on the published 0.4.0, ~40 on long generations), 17.5 GiB parked - room/plan.js: bestFit() (picker preselects the biggest model that fits) - cli/README.md "Claude Code": its prompt is 15-20k tokens; use the MoE or 27B - room/pledge.js (phone caps) @@ -30,7 +28,6 @@ Pick by how much memory the room has. The header shows it as `N GB pooled`, and | Qwen3 1.7B | 5.6 GB (4 GB at 8K) | 1.83 GB | 16K (8K when the room is short) | | Qwen3.8 27B | 17 GB | 16.06 GB | 16K (up to 64K) | | Qwen3.6 35B MoE | 22.5 GB | 20.84 GB | 32K (up to 128K) | -| Qwen3.5 122B MoE (preview: `?dev=1`, or the terminal) | 73 GB | 72.52 GB | 32K (up to 128K) | | Qwen3.8-Flash-Next (**experimental**; `POOLED_DEV=1` for `pooled host` / `list`) | 72 GB, or about 8 GB on one tab with its experts on disk | 100.6 GB (71.8 GB of layers, 28.8 GB n-gram table) | 64K (the picker offers 8K, 32K and 64K; 128K in progress) | The download is split across the devices that run the model, so each one fetches only its part. @@ -45,8 +42,6 @@ The 1.7B opens with a 16K context when the room has 5.6 GB for it. With less, do - **Claude Code or Codex through the [Serve API](/docs/serve):** the MoE or the 27B. Their prompts alone are too long for the 1.7B. - **One PC with a 12 GB GPU:** the MoE from the terminal, with `pooled host qwen3.6-35b-moe --here`. It keeps the experts the GPU can't hold in system RAM (about 19 GB of them) and runs at about 38-45 tok/s on an RTX 5070. See [MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram). -In the terminal, `pooled host` and `pooled pull` also list the **Qwen3.5 122B MoE**: about 73 GB across the room and a 72.52 GB download. A Mac Studio lending 25 GB and an RTX 5070 PC keeping 42 GB of its experts in RAM run it at about 17 tok/s. The room page shows it only with `?dev=1`. - When a room's GPUs and RAM together still fall short of a MoE model, a computer started with `--disk` can read some of its experts from the model's files on its SSD instead. The room does this only as a last resort, and it is slower. See [MoE experts read from disk](/docs/terminal#moe-experts-read-from-disk). A browser tab has its own disk tier for Qwen3.8 Flash (below). Phones add little memory: 0.5 to 2 GB each. See [Phones and tablets](/docs/phones). diff --git a/docs-site/src/content/docs/openclaw.mdx b/docs-site/src/content/docs/openclaw.mdx index c2681b45..5a98c415 100644 --- a/docs-site/src/content/docs/openclaw.mdx +++ b/docs-site/src/content/docs/openclaw.mdx @@ -94,7 +94,7 @@ The room needs at least a 16k context, because OpenClaw's own instructions and t ## Which model -Use the **Qwen3.6 35B MoE** (about 23 GB across the room). Onboarding preselects it when this device's memory holds it, the Qwen3.8 27B (about 20 GB) when only that fits, and the MoE again when neither fits on this device alone (the room then waits for more devices). A device that joins with `pooled join` (`@pooled/cli` 0.4.0 or newer) on a discrete GPU can make up a short room by keeping some of the MoE's experts in system RAM, more slowly ([MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram)). The gateway itself holds its layers whole. The Qwen3.5 122B MoE is in `pooled host` and `pooled pull` from 0.4.0, not in OpenClaw's onboarding. +Use the **Qwen3.6 35B MoE** (about 23 GB across the room). Onboarding preselects it when this device's memory holds it, the Qwen3.8 27B (about 20 GB) when only that fits, and the MoE again when neither fits on this device alone (the room then waits for more devices). A device that joins with `pooled join` (`@pooled/cli` 0.4.0 or newer) on a discrete GPU can make up a short room by keeping some of the MoE's experts in system RAM, more slowly ([MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram)). The gateway itself holds its layers whole. :::caution[Small models] Small models struggle with OpenClaw's long prompts and tools: expect slow turns and tool loops. Use the 35B MoE if your devices can hold it. diff --git a/docs-site/src/content/docs/reference/cli.mdx b/docs-site/src/content/docs/reference/cli.mdx index 605324c8..5db8a89b 100644 --- a/docs-site/src/content/docs/reference/cli.mdx +++ b/docs-site/src/content/docs/reference/cli.mdx @@ -91,4 +91,4 @@ The repo's `npm run serve` also uses 8080. Pass `--port 8081` and change the bas ## Version -`pooled --version` prints the version you have; `npx @pooled/cli@latest` runs the newest. From 0.4.0, `pooled host` and `pooled join` take `--ram`, which lets a discrete GPU keep a MoE model's experts in system RAM ([MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram)), and `pooled pull` / `pooled host` list the Qwen3.5 122B MoE. See the [changelog](/docs/reference/changelog). +`pooled --version` prints the version you have; `npx @pooled/cli@latest` runs the newest. From 0.4.0, `pooled host` and `pooled join` take `--ram`, which lets a discrete GPU keep a MoE model's experts in system RAM ([MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram)). See the [changelog](/docs/reference/changelog). diff --git a/docs-site/src/content/docs/reference/models.mdx b/docs-site/src/content/docs/reference/models.mdx index e87f74b5..294801f5 100644 --- a/docs-site/src/content/docs/reference/models.mdx +++ b/docs-site/src/content/docs/reference/models.mdx @@ -12,9 +12,8 @@ Sources (origin/main 40bb0fa): KV bytes per position (27B 64 KB f16, MoE 20 KB), MAX_NEW 400, MAX_NEW_THINKING 1200, MIN_ROOM 32 - room.js:3458 ANSWER_LEN { short: 150, normal: MAX_NEW, long: 1200 } - docs/models.md (engines, layer layout, local test layout, adding a model), README.md Models table -- 0.4.0 (origin/main ed31899): CTX maxes (27B 65536, MoE 131072, 122B 32768/131072), the 122B MODELS/FILES/NEED_GB 72 (#316), - cli/lib/lend.js hostable() lists it for pooled pull / host; expert offload #319/#320/#323 (numbers: docs/bench-log.md 2026-10-02 - phase 2b, the maintainer's Mac Studio + RTX 5070 122B run) +- 0.4.0 (origin/main ed31899): CTX maxes (27B 65536, MoE 131072); expert offload #319/#320/#323 (numbers: docs/bench-log.md 2026-10-02 + phase 2b) - MoE 40 layers: docs/bench-log.md ("35B-A3B MoE (40 layers"); 10 attention layers from room/models.js comment - 27B 64 layers (48 DeltaNet + 16 attention): docs/models.md */} @@ -44,7 +43,6 @@ Pooled ships no weights. Each device downloads only its share of the layers from | Qwen3 1.7B | [`Qwen/Qwen3-1.7B-GGUF`](https://huggingface.co/Qwen/Qwen3-1.7B-GGUF) `Qwen3-1.7B-Q8_0.gguf` | | Qwen 3.8 27B | [`unsloth/Qwen3.8-27B-GGUF`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) `Qwen3.8-27B-Q4_0.gguf` | | Qwen 3.6 35B MoE | [`bartowski/Qwen_Qwen3.6-35B-A3B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF) `Qwen_Qwen3.6-35B-A3B-Q4_0.gguf` | -| Qwen 3.5 122B MoE | [`bartowski/Qwen_Qwen3.5-122B-A10B-GGUF`](https://huggingface.co/bartowski/Qwen_Qwen3.5-122B-A10B-GGUF) `Qwen_Qwen3.5-122B-A10B-Q4_0-0000{1,2}-of-00002.gguf` (two files) | The Qwen3 dense models also fetch `config.json` and `tokenizer.json` from the matching `Qwen/Qwen3-*` repository. The 27B and the MoE read everything from the GGUF. @@ -72,20 +70,19 @@ These stay in `room/models.js` for tests. The picker shows them with [`?dev=1`]( | Qwen3 0.6B | GGUF Q8_0 | `DenseEngine` | 0.8 GB | 2048 | | Qwen3 4B | GGUF Q8_0 | `DenseEngine` | 4.6 GB | 2048 | | SmolLM2 135M | safetensors bf16 | `DenseEngine` | 0.6 GB | 2048 | -| Qwen 3.5 122B MoE (122B-A10B) | GGUF Q4_0, two files, 72.52 GB | `Qwen35Engine` | 72 GB | 32768 / 131072 | | Qwen3.8-Flash-Next (experimental) | GGUF Q4_0, three files, 100.6 GB (71.8 GB of layers, a 28.8 GB n-gram table) | `Qwen4ExpEngine` | 72 GB, plus the n-gram table on the device holding layer 1 | 65536 / 65536 | -Qwen3 0.6B is the golden-test model. Qwen3.8-Flash-Next is experimental: it is a row in the picker for everyone (`room/models.js` `DISK_COPY`) and needs `POOLED_DEV=1` for `pooled host` / `pooled list`; its greedy output matches llama.cpp's golden ids. A tab can keep its experts on the SSD (OPFS, after its user allows it), and a room node can park them in RAM or read them from its files. Rooms load it with QSA at up to 64K tokens of context (f16 K/V, checked against llama.cpp at 2K, 8K, 16K, 32K and 64K); 128K is in progress. The 122B MoE has 48 layers and 256 experts, 8 active per token (about 10B of the 122B). Besides `?dev=1`, it is in the terminal's list: `pooled pull` and `pooled host` (from `@pooled/cli` 0.4.0) offer it with what it needs. +Qwen3 0.6B is the golden-test model. Qwen3.8-Flash-Next is experimental: it is a row in the picker for everyone (`room/models.js` `DISK_COPY`) and needs `POOLED_DEV=1` for `pooled host` / `pooled list`; its greedy output matches llama.cpp's golden ids. A tab can keep its experts on the SSD (OPFS, after its user allows it), and a room node can park them in RAM or read them from its files. Rooms load it with QSA at up to 64K tokens of context (f16 K/V, checked against llama.cpp at 2K, 8K, 16K, 32K and 64K); 128K is in progress. ## Experts in system RAM -On the 35B and the 122B MoE, a `pooled host` or `pooled join` device with a discrete GPU can keep some layers' experts in system RAM when the room's pledges can't hold the model (`--ram`). It copies the experts each token uses into a cache on its GPU. One RTX 5070 (12 GB) runs the 35B alone at about 38-45 tok/s this way, and a Mac Studio (25 GB) with an RTX 5070 PC keeping 42 GB of experts in RAM runs the 122B at about 17 tok/s. Unified memory and browsers don't do this. See [MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram). +On the 35B MoE, a `pooled host` or `pooled join` device with a discrete GPU can keep some layers' experts in system RAM when the room's pledges can't hold the model (`--ram`). It copies the experts each token uses into a cache on its GPU. One RTX 5070 (12 GB) runs the 35B alone at about 38-45 tok/s this way. Unified memory and browsers don't do this. See [MoE experts in system RAM](/docs/terminal#moe-experts-in-system-ram). ## Engines | Engine | Models | |---|---| -| `Qwen35Engine` | Qwen 3.8 27B, Qwen 3.6 35B MoE (`qwen35moe`), Qwen3.5-122B-A10B | +| `Qwen35Engine` | Qwen 3.8 27B, Qwen 3.6 35B MoE (`qwen35moe`) | | `DenseEngine` | Qwen3 0.6B, 1.7B, 4B, SmolLM2 135M | | `Qwen4ExpEngine` | Qwen3.8-Flash-Next (llama.cpp `qwen4exp`): 512 experts, 10 active, hyper-connections, a per-layer n-gram table and QSA (`engine/qwen4exp.js`; rooms through `room/flash.js`) | diff --git a/docs-site/src/content/docs/reference/url-options.mdx b/docs-site/src/content/docs/reference/url-options.mdx index f1173fa7..3f6acce8 100644 --- a/docs-site/src/content/docs/reference/url-options.mdx +++ b/docs-site/src/content/docs/reference/url-options.mdx @@ -52,7 +52,6 @@ Supported and stable. | Qwen3 1.7B | 16384 (8192 when the room is short of memory) | 16384 | | Qwen 3.8 27B | 16384 | 65536 | | Qwen 3.6 35B MoE | 32768 | 131072 | -| Qwen3.5 122B MoE (`?dev=1`) | 32768 | 131072 | | Qwen3.8-Flash-Next (experimental) | 65536 (a room short of memory opens at the longest proven length it holds: 32768, 16384, 8192 or 2048) | 65536 (128K is in progress) | | Other models (tests and `?dev=1`) | 2048 (8192 on the hybrid engine) | same | diff --git a/docs-site/src/content/docs/roadmap.mdx b/docs-site/src/content/docs/roadmap.mdx index 540c1979..dc1b391a 100644 --- a/docs-site/src/content/docs/roadmap.mdx +++ b/docs-site/src/content/docs/roadmap.mdx @@ -49,7 +49,7 @@ Issues get the `area:` label of their section. As of October 11, 2026, every roadmap line was checked against the code: - **`@pooled/cli` 0.4:** `pooled serve` with OpenAI and Anthropic endpoints and tool calling, `pooled host` / `join` / `chat` on a native peer, and the OpenClaw plugin ([#4](https://github.com/Nehanth/pooled/issues/4), [#11](https://github.com/Nehanth/pooled/issues/11)). -- **Bigger models:** the Qwen3.5 122B, and MoE experts kept in system RAM (`--ram`), so a Mac Studio and an RTX 5070 PC run the 122B at about 17 tok/s. +- **Bigger models:** MoE experts kept in system RAM (`--ram`), so one RTX 5070 PC runs the 35B MoE alone at about 38-45 tok/s. - **Security:** channel-bound join auth, where keys and passes are proved, never sent. - **Networks:** the TURN relay, in the browser and now the CLI ([#1](https://github.com/Nehanth/pooled/issues/1)). - **Speed and sessions:** the prefill kernels ([#2](https://github.com/Nehanth/pooled/issues/2)); checkpoints on disk and prefix reuse across agent turns ([#69](https://github.com/Nehanth/pooled/issues/69), [#73](https://github.com/Nehanth/pooled/issues/73)); GPU sampling ([#26](https://github.com/Nehanth/pooled/issues/26)); the GEMV 8-row fix ([#47](https://github.com/Nehanth/pooled/issues/47)). diff --git a/docs-site/src/content/docs/terminal.mdx b/docs-site/src/content/docs/terminal.mdx index f01f8c03..b66f53e4 100644 --- a/docs-site/src/content/docs/terminal.mdx +++ b/docs-site/src/content/docs/terminal.mdx @@ -14,8 +14,7 @@ Sources (feat/pooled-join 88dc35a, PR #270, stacked on #266; the room page's gat - `node cli/bin/pooled.js host --help`, `join --help`, `chat --help`, `--help` (copied below as printed, 2026-09-30) - `node cli/bin/pooled.js host --model list` (the five model keys) - 0.4.0 (origin/main ed31899): --ram and expert offload (#319, #320, #323: cli/lib/lend.js parseRam/ramRule, RAM_KEEP_GB 16; - room/plan.js; 35B on the RTX 5070 alone about 38-45 tok/s (37.5-44.9 measured on the published 0.4.0) and the 122B Spark/Mac+PC rooms: docs/bench-log.md 2026-10-02 and the - maintainer's Mac Studio + RTX 5070 run, ~17 tok/s), the 122B (#316; NEED_GB 72 -> the picker's "needs 73 GB"; FILES 72.5 GB), + room/plan.js; 35B on the RTX 5070 alone about 38-45 tok/s (37.5-44.9 measured on the published 0.4.0): docs/bench-log.md 2026-10-02), help blocks re-copied from `node cli/bin/pooled.js --help` - --disk (roadmap/32 steps 3-5: cli/lib/lend.js diskRule, DISK_RAM_MIN_GB 2; room/plan.js DISK_RAM_MIN, offloadNeedDisk; engine/expert_t1.js; packages/room-node/shard.js expertReader): help blocks re-copied from `node cli/bin/pooled.js --help` @@ -179,7 +178,7 @@ On a discrete GPU it then allocates that much for a moment, to make sure it is r ### MoE experts in system RAM -From `@pooled/cli` 0.4.0, a computer with a discrete GPU can hold more of a mixture-of-experts model (the 35B MoE or the 122B MoE) than its GPU pledge. It keeps some layers' experts in system RAM and copies the ones each token uses into a cache on the GPU. The room does this only when the pledges alone can't hold the model. It is slower than holding the layers whole, and while a device does it the room decodes without speculation, which is faster there. +From `@pooled/cli` 0.4.0, a computer with a discrete GPU can hold more of a mixture-of-experts model (the 35B MoE) than its GPU pledge. It keeps some layers' experts in system RAM and copies the ones each token uses into a cache on the GPU. The room does this only when the pledges alone can't hold the model. It is slower than holding the layers whole, and while a device does it the room decodes without speculation, which is faster there. - `--ram 24` lets the experts use up to 24 GB of RAM. `--ram 0` turns it off. - The default is the total RAM less 16 GB, never more than is free at start less 4 GB, and at most 64 GB. @@ -190,7 +189,6 @@ The header shows `RAM up to N GB`, and the host's log says which layers each dev | Room | Speed | |---|---| | Qwen3.6 35B MoE on one RTX 5070 (12 GB): `pooled host qwen3.6-35b-moe --here` | about 38-45 tok/s | -| Qwen3.5 122B MoE on a Mac Studio (lends 25 GB) and an RTX 5070 PC (the rest of the layers, 42 GB of experts in RAM) | about 17 tok/s, matching llama.cpp | ### MoE experts read from disk @@ -376,13 +374,10 @@ qwen3-1.7b Qwen3 1.7B · Q8 qwen3-4b Qwen3 4B · Q8 qwen3.8-27b Qwen3.8 27B · Q4 qwen3.6-35b-moe Qwen3.6 35B MoE · Q4 -qwen3.5-122b-moe Qwen3.5 122B MoE · Q4 ``` With `POOLED_DEV=1`, the list also has `qwen3.8-flash` (Qwen3.8-Flash-Next · Q4, a 100.6 GB download in three files: 71.8 GB of layers and a 28.8 GB n-gram table). It is experimental (the CLI calls it a preview): rooms run it at up to 64K tokens of context. The CLI on npm doesn't have Flash's newest code yet, so run the CLI from the repo to lend or host it (see the [CLI README](https://github.com/Nehanth/pooled/blob/main/cli/README.md#qwen38-flash-experimental-run-the-cli-from-the-repo)). `pooled join` needs no flag to lend to a room that runs it, and `pooled pull qwen3.8-flash` / `pooled rm qwen3.8-flash` take its full name without `POOLED_DEV=1`. -The Qwen3.5 122B MoE is a 72.5 GB download (two files) and needs about 73 GB across the room. It is for a room of big machines, or one with a PC that keeps its experts in RAM. The room page lists it only with `?dev=1`. - ### pooled host --help ```text diff --git a/docs/models.md b/docs/models.md index b635836c..01c807a8 100644 --- a/docs/models.md +++ b/docs/models.md @@ -8,12 +8,12 @@ Pooled ships no weights. Browsers fetch tensors by HTTP range request from publi |---|---|---|---| | Qwen 3.8 27B | GGUF Q4_0 (~15 GB), includes the `nextn` draft layer | `Qwen35Engine` | 64 layers: 48 Gated DeltaNet + 16 attention; MTP speculation; 16K context by default, up to 64K | | Qwen 3.6 35B MoE (Qwen3.6-35B-A3B) | GGUF Q4_0 (bartowski) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active per token (~3B of 35B); router, experts and shared expert on the GPU; 32K context by default, up to 128K | -| Qwen3.5-122B-A10B MoE | GGUF Q4_0 (bartowski, two files, 72.5 GB) | `Qwen35Engine` (`qwen35moe`) | 256 experts, 8 active (~10B of 122B); in `pooled pull` / `list` / `host` from `@pooled/cli` 0.4.0, on the room page only with `?dev=1`; 32K context by default, up to 128K | +| Qwen3.5-122B-A10B MoE | GGUF Q4_0 (bartowski, two files, 72.5 GB) | `Qwen35Engine` (`qwen35moe`) | **Developer only, hidden from users**: in the room page's `?dev=1` select and, with `POOLED_DEV=1`, in `pooled list` / `host` / `pull` (pull / rm take its full key without it); `tests/test_moe.js MODEL=122b`. 256 experts, 8 active (~10B of 122B); 32K context by default, up to 128K | | Qwen3.8 Flash Next (**experimental**) | GGUF Q4_0 (bartowski, three files, ~100.6 GB, of which a 28.8 GB n-gram table) | `Qwen4ExpEngine` (`engine/qwen4exp.js`, rooms via `room/flash.js`) | 125B, 512 experts, 10 active per token (~6B); most experts stay on disk and stream into a GPU cache (disk tier); context up to 64K, tested against llama.cpp; 128K in progress; from the command line, run the CLI from the repo ([cli/README.md](../cli/README.md#qwen38-flash-experimental-run-the-cli-from-the-repo)); in the room picker | | Qwen3 4B / 1.7B / 0.6B | GGUF Q4_0 / Q8_0 | `DenseEngine` | dense; 0.6B is the golden-test model | | SmolLM2 135M | safetensors f32 | `DenseEngine` | smallest demo; quantized to Q8 at load | -The room's model list lives in `room/models.js`: `MODELS` (URLs), `NEED_GB` (memory the whole model needs), `PICKER` (what the picker offers: Qwen3 1.7B, Qwen 3.8 27B, Qwen 3.6 35B MoE, plus the models in `DISK_COPY`, which keep a copy on each device's disk: Qwen3.8-Flash-Next; the others, the 122B included, stay for tests, the CLI and `?dev=1`) and `CTX` (default and largest context per model). Architectures sharing Qwen 3.5/3.6/3.8's hybrid layout (e.g. Qwen 3.5 2B/9B) need only entries there. +The room's model list lives in `room/models.js`: `MODELS` (URLs), `NEED_GB` (memory the whole model needs), `PICKER` (what the picker offers: Qwen3 1.7B, Qwen 3.8 27B, Qwen 3.6 35B MoE, plus the models in `DISK_COPY`, which keep a copy on each device's disk: Qwen3.8-Flash-Next; the others, the developer-only 122B included, stay for tests and `?dev=1`) and `CTX` (default and largest context per model). Architectures sharing Qwen 3.5/3.6/3.8's hybrid layout (e.g. Qwen 3.5 2B/9B) need only entries there. ## Local layout for tests and benchmarks diff --git a/docs/openclaw.md b/docs/openclaw.md index d633a9a9..58853456 100644 --- a/docs/openclaw.md +++ b/docs/openclaw.md @@ -79,8 +79,7 @@ Use the Qwen3.6 35B MoE (about 23 GB across the room). Onboarding preselects it memory holds it, the Qwen3.8 27B (about 20 GB) when only that fits, and the MoE again when neither fits on this device alone (the room then waits for more devices). A `pooled join` device (`@pooled/cli` 0.4.0 or newer) on a discrete GPU can make up a short room by keeping some experts in system RAM -(`--ram`); the gateway holds its layers whole. The Qwen3.5 122B MoE is in `pooled host`, not in -OpenClaw's onboarding. Picking the 1.7B, or joining a room that +(`--ram`); the gateway holds its layers whole. Picking the 1.7B, or joining a room that runs it, shows: "Small models struggle with OpenClaw's long prompts and tools: expect slow turns and tool loops. Use the 35B MoE if your devices can hold it." Measured on a DGX Spark, the OpenClaw gateway hosting and a second device (`pooled join`) holding the diff --git a/docs/releases/3.0.md b/docs/releases/3.0.md index 6102664c..97ec6085 100644 --- a/docs/releases/3.0.md +++ b/docs/releases/3.0.md @@ -37,7 +37,6 @@ Flash rooms run at up to 64K tokens of context (#396), checked against llama.cpp - The speed panel shows live tok/s, each device, the time per hop and the expert cache's hit rate (#353, #358). ### Other models -- **Qwen3.5 122B MoE** in `pooled pull` / `list` / `host`, ~73 GB across the room. A Mac Studio plus an RTX 5070 PC keeping its experts in RAM runs it at ~17 tok/s. On the room page with `?dev=1`. - Qwen3 1.7B opens at 16K context (8K when the room is short); the 35B MoE goes up to 128K, the 27B up to 64K. - The tokenizer uses Qwen's own pre-tokenizer split, checked against llama.cpp. diff --git a/docs/tech-stack.md b/docs/tech-stack.md index 27ae48c2..1edf96c4 100644 --- a/docs/tech-stack.md +++ b/docs/tech-stack.md @@ -29,7 +29,7 @@ What Pooled is built from, and why each piece was chosen. There is no build step ## Models -Qwen 3.8 27B (hybrid Gated-DeltaNet + attention, `qwen35` architecture family), Qwen 3.6 35B MoE and Qwen3.5-122B MoE (`qwen35moe`), Qwen3 0.6B/1.7B/4B (dense), SmolLM2 135M. See [models.md](models.md). +Qwen 3.8 27B (hybrid Gated-DeltaNet + attention, `qwen35` architecture family), Qwen 3.6 35B MoE (`qwen35moe`), Qwen3 0.6B/1.7B/4B (dense), SmolLM2 135M. See [models.md](models.md). ## Things deliberately not used diff --git a/packages/openclaw/README.md b/packages/openclaw/README.md index fbf70dae..de60fc60 100644 --- a/packages/openclaw/README.md +++ b/packages/openclaw/README.md @@ -101,8 +101,7 @@ tools) once, then keeps it. Onboarding preselects the MoE when this device can hold it, and the 27B when only that fits. A room can hold more than any one device: it waits for more devices. A `pooled join` device (`@pooled/cli` 0.4.0 or newer) with a discrete GPU can make up a short room by keeping some of the MoE's experts in system RAM -(`--ram`), more slowly; the gateway holds its layers whole. The Qwen3.5 122B MoE is in `pooled host`, -not in OpenClaw's onboarding. +(`--ram`), more slowly; the gateway holds its layers whole. ## Requirements diff --git a/packages/room-node/README.md b/packages/room-node/README.md index 0f439a98..04b0ca96 100644 --- a/packages/room-node/README.md +++ b/packages/room-node/README.md @@ -47,7 +47,7 @@ const node = await joinRoom("K7QX", { pledgeGB: 16 }); | `ask` | (createRoom, with `gate`) false lets anyone with the code in without asking (`pooled host --allow-all`); default true | | `key`, `pass` | (joinRoom) the invite key from a room link, and a pass from an earlier admit: sent only to the host (`hello {join: 1, key, pass}`). The node waits in a gated host's lobby (`lobby`, then `admitted`; `node.pass` is what the host gave it) and walks straight into a host from before the gate | | `pledgeGB` | GPU memory this device lends; default half its largest buffer (1 to 64). It may say up to 128 (`PLEDGE_MAX_GB`), but the host's deal counts more than 64 only for Qwen3.8-Flash-Next (`room/pledge.js` `maxPledgeGB`) | -| `ctx` | context to ask for, clamped by `room/models.js` CTX (1.7B 16k, 27B 64k, MoE 128k, 122B MoE 128k, Qwen3.8-Flash-Next 64k); default the largest for the qwen35 models (MoE 128k, 27B 64k) and 16k for the 1.7B (8k when the room's pledges are short for 16k and no `ctx` was asked: room/models.js `pickCtx`), then lowered to what the smallest GPU binding limit in the room holds (`maxBindMB` in each hello; none counts as 128 MiB). Every device gets it with its `ai-load` | +| `ctx` | context to ask for, clamped by `room/models.js` CTX (1.7B 16k, 27B 64k, MoE 128k, Qwen3.8-Flash-Next 64k); default the largest for the qwen35 models (MoE 128k, 27B 64k) and 16k for the 1.7B (8k when the room's pledges are short for 16k and no `ctx` was asked: room/models.js `pickCtx`), then lowered to what the smallest GPU binding limit in the room holds (`maxBindMB` in each hello; none counts as 128 MiB). Every device gets it with its `ai-load` | | `ckptDisk` | disk copies of the pinned checkpoints (`ckptdisk.js`), so a restarted room resumes its system prompt + tools instead of reading them again: a `CkptDisk`, or `false` for none. Default `~/.pooled/cache/ckpt` (0700; files 0600), at most 4 GB, least recently used out first; `POOLED_CKPT_DISK=0` turns it off, `POOLED_CKPT_GB` / `POOLED_CKPT_DIR` set the cap and the folder. Every device (host and workers) keeps its own layers' part. A copy is used only for exactly the room that made it (model and model file, context, KV mode, every device's layer range and engine state signature, the prompt's token ids); otherwise the host prefills as usual. The files hold prompt-derived state (the KV cache of the system prompt): treat them like the prompts, delete the folder to clear them | | `ckpt` | the host's checkpoints (`ckpt.js`): `{ answers, pins, minPin, turns }` (default 4 answer and turn checkpoints, 4 pinned prefixes, pin a fixed start of 1024+ tokens, turn checkpoints on), or `false` for none | | `modelDir` | local model files: `/` as `pooled pull` keeps them (`qwen3-1.7b/Qwen3-1.7B-Q8_0.gguf`), or the layout of `source.js` LOCAL (`qwen17/model.gguf`, `q36moe/Qwen_Qwen3.6-35B-A3B-Q4_0.gguf`, ...); else HTTP range reads of the model's URL, as the page makes | @@ -99,7 +99,7 @@ const node = await joinRoom("K7QX", { pledgeGB: 16 }); node packages/room-node/join.mjs K7QX --gb 12 --name mac-studio [--ram 32] [--signal host:port] [--models dir] [--state status.json] ``` -`--ram N` (`ramGB` in `createRoom` / `joinRoom`): expert offload. The device tells the room it can keep a MoE model's routed experts in N GB of system RAM (`meta.offload`, `meta.ramGB`); when the pledges can't hold the model, the host deals it layers past its pledge with `ai-load {offload: {lo, hi, vramBytes, ramBytes}}`, and `shard.js` parks those layers' experts in an `ExpertStore` (engine/expert_store.js) with a GPU cache of `vramBytes`. Only for a discrete GPU (`pooled` sets it from the memory rule); 0 (the default here) never offloads. On unified memory (Apple silicon, a GB10) `probeMeta` refuses it whatever `ramGB` says (`mem`: `detectMemory()`'s answer, else it asks the OS). The deal sizes the parked experts exactly as `ExpertStore` parks them (`room/models.js expertsOf`: Q4_1 experts park as Q8), and a load whose experts would park more than `ramGB` fails instead of coming online. The deal keeps room on such a device (`room/plan.js offloadSpare`): `vramBytes` leaves part of the pledge for the engine's working buffers and the host's checkpoints, and `offload.ramSpare` is RAM kept free past the experts (the experts park within `ramGB - ramSpare`). A deal with offload decodes without speculation by default (plain decoding is faster there); `offloadSpec: true` (or `POOLED_OFFLOAD_SPEC=1`) forces it on, `false` off. A split GGUF (the 122B: two files) is read as one model, from `//` when every file is there, else from Hugging Face, each tensor from its own file. +`--ram N` (`ramGB` in `createRoom` / `joinRoom`): expert offload. The device tells the room it can keep a MoE model's routed experts in N GB of system RAM (`meta.offload`, `meta.ramGB`); when the pledges can't hold the model, the host deals it layers past its pledge with `ai-load {offload: {lo, hi, vramBytes, ramBytes}}`, and `shard.js` parks those layers' experts in an `ExpertStore` (engine/expert_store.js) with a GPU cache of `vramBytes`. Only for a discrete GPU (`pooled` sets it from the memory rule); 0 (the default here) never offloads. On unified memory (Apple silicon, a GB10) `probeMeta` refuses it whatever `ramGB` says (`mem`: `detectMemory()`'s answer, else it asks the OS). The deal sizes the parked experts exactly as `ExpertStore` parks them (`room/models.js expertsOf`: Q4_1 experts park as Q8), and a load whose experts would park more than `ramGB` fails instead of coming online. The deal keeps room on such a device (`room/plan.js offloadSpare`): `vramBytes` leaves part of the pledge for the engine's working buffers and the host's checkpoints, and `offload.ramSpare` is RAM kept free past the experts (the experts park within `ramGB - ramSpare`). A deal with offload decodes without speculation by default (plain decoding is faster there); `offloadSpec: true` (or `POOLED_OFFLOAD_SPEC=1`) forces it on, `false` off. A split GGUF (Qwen3.8-Flash-Next: three files) is read as one model, from `//` when every file is there, else from Hugging Face, each tensor from its own file. `expertDisk: true` (`pooled --disk`): expert offload's disk tier (roadmap/32). The hello adds `meta.disk` and `meta.local` (the models whose every file is in `modelDir`: `source.js localModels`), only on a device that offloads. When parking can't hold the model either, the host deals such a device `offload.diskLo` and `offload.diskRamBytes` for a model it has: layers `[diskLo, hi)` keep their experts in the files, `shard.js` reads them when a token picks them (`expert_disk.js`'s worker threads; the engine's in-thread reader if no worker can run) through a RAM cache of `diskRamBytes` (`engine/expert_t1.js`), and `offloadOk` checks parked + spare + cache against `ramGB`. Freeing the layers closes the reader and the cache. diff --git a/room.js b/room.js index 714ffb14..d644cfc7 100644 --- a/room.js +++ b/room.js @@ -5256,6 +5256,12 @@ async function aiOnData(from, d) { sendTo(from, { t: "ai-error", message: remote }); return; } + // a model this page's catalogue doesn't have (a host on another version): refused out loud, as packages/room-node does + if (d.t === "ai-load" && !MODELS[d.model]) { + log("room", `refused layers of unknown model ${String(d.model).slice(0, 40)}`); + sendTo(from, { t: "ai-error", load: 1, message: `${myName} refused the layers: unknown model ${String(d.model).slice(0, 40)}` }); + return; + } // returned hidden states are only accepted from the end of the chain if ((d.t === "ai-hiddenret" || d.t === "ai-hiddenret-b") && from !== ai.chain[ai.chain.length - 1]) return; switch (d.t) {