Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ All notable changes to Pooled (called SwarmLLM before September 2026). Format fo

## [3.0.0] - 2026-10-16

**Run a 125B model from your SSD, in a browser tab, and pool devices to make it faster.** Pooled 3.0 runs Qwen3.8 Flash (**experimental**), a 125B mixture-of-experts model with about 6B active per token. It is far bigger than most GPUs' memory, so each device keeps its share of the experts on its SSD (browser storage for a tab, the model files for `pooled join`), caches recently used experts on its GPU (routing repeats: about 17% of a layer's experts serve about 77% of lookups), and reads the rest from disk as tokens need them. Disk is the slow part; every device that joins holds part of the model in its memory, so less comes from disk and it gets faster: a Mac Studio alone in Chrome decodes about 8 tok/s, and with an RTX 5070 PC joined about 10-12 tok/s, depending on the network. Flash rooms run at up to 64K context (128K is in progress). 3.0 also brings the system-RAM and SSD expert tiers in the terminal, the Qwen3.5 122B MoE, `pooled serve`, channel-bound join auth and the default TURN relay. Release notes and the demo video: [docs/releases/3.0.md](docs/releases/3.0.md).
**Run a 125B model from your SSD, in a browser tab, and pool devices to make it faster.** Pooled 3.0 runs Qwen3.8 Flash (**experimental**), a 125B mixture-of-experts model with about 6B active per token. It is far bigger than most GPUs' memory, so each device keeps its share of the experts on its SSD (browser storage for a tab, the model files for `pooled join`), caches recently used experts on its GPU (routing repeats: about 17% of a layer's experts serve about 77% of lookups), and reads the rest from disk as tokens need them. Disk is the slow part; every device that joins holds part of the model in its memory, so less comes from disk and it gets faster: a Mac Studio alone in Chrome decodes about 8 tok/s, and with an RTX 5070 PC joined about 10-12 tok/s, depending on the network. Flash rooms run at up to 64K context (128K is in progress). 3.0 also brings the system-RAM and SSD expert tiers in the terminal, `pooled serve`, channel-bound join auth and the default TURN relay. Release notes and the demo video: [docs/releases/3.0.md](docs/releases/3.0.md).

Not in 3.0: the CLI on npm (`@pooled/cli` 0.4.1) doesn't have Flash's newest code yet; to lend or host Flash from the terminal, run the CLI from the repo ([cli/README.md](cli/README.md#qwen38-flash-experimental-run-the-cli-from-the-repo)). Flash MTP speculative decoding (#352) is not part of 3.0.

Expand All @@ -33,9 +33,9 @@ Not in 3.0: the CLI on npm (`@pooled/cli` 0.4.1) doesn't have Flash's newest cod
- **An explicit 128 GB pledge cap for Qwen3.8-Flash-Next only** (#356, #369; `room/pledge.js` `PLEDGE_MAX_GB = 128`, `maxPledgeGB`): one big unified-memory machine lending from the terminal (`pooled host` / `pooled join`) can hold Flash alone; a browser tab still lends at most 64 GB. Every other model still counts at most 64 GB per device, and the default memory rule is unchanged. `pooled join --gb max` may still say up to 128 GB, because a joiner doesn't know the room's model yet; the host counts 64 of it for non-Flash models.
- **The speed panel** (#353, #358): live tok/s, device cards, per-hop timing from room nodes and the expert-cache hit rate.
- **Expert offload's per-layer cut in Chrome: 2.9 ms → 0.5 ms** (#351): a map kick (on by default, Chrome only), a pipelined cut (`cutPipe`, opt-in) and speculative windows.
- **`--disk`: MoE experts read from the SSD when the GPU and RAM are short (roadmap 32, steps 3-5; the 35B, the 122B and Flash, #368).** `pooled join --disk` and `pooled host --disk` (with `--here` too) let a computer with a discrete GPU and `--ram` leave some layers' routed experts in the model's files and read them when a token picks them, through a RAM cache of per-expert upload buffers (at least 2 GB of the `--ram`; segmented LRU, an expert is cached on its second miss). The room uses it only when parking experts in RAM can't hold the model, and a room that fits without it deals exactly as before. Outputs stay bit-exact (the 35B with every layer on disk gives main's hashes).
- **`--disk`: MoE experts read from the SSD when the GPU and RAM are short (roadmap 32, steps 3-5; the 35B and Flash, #368).** `pooled join --disk` and `pooled host --disk` (with `--here` too) let a computer with a discrete GPU and `--ram` leave some layers' routed experts in the model's files and read them when a token picks them, through a RAM cache of per-expert upload buffers (at least 2 GB of the `--ram`; segmented LRU, an expert is cached on its second miss). The room uses it only when parking experts in RAM can't hold the model, and a room that fits without it deals exactly as before. Outputs stay bit-exact (the 35B with every layer on disk gives main's hashes).
- **The CLI uses the TURN relay too.** `pooled host`, `pooled join`, `pooled chat` and `pooled serve` (and the OpenClaw plugin's room node) used to try direct links only (public STUN), so two devices behind strict NATs or CGNAT could not reach each other from the terminal, though the room page could through pooled.run's relay. They now fetch the same short-lived relay credentials (`cli/lib/relay.js`) and use the relay only when no direct path works, refreshing the credentials at 3/4 of their lifetime. `/api/turn` lets a request with no `Origin` through when it names itself in `X-Pooled-Client` (a browser on another site always sends `Origin`, so other pages still can't mint credentials; the per-IP limit applies). `POOLED_RELAY=0` / `1` turns the relay off / forces it, `POOLED_TURN` (+ `POOLED_TURN_USER`, `POOLED_TURN_CRED`) uses your own, `POOLED_TURN_API` another deployment's. Tested with every link forced through Cloudflare's relay: `pooled host` and `pooled chat` on a GB10 answered over relayed links only.
- **`@pooled/cli` 0.4.0 and `@pooled/openclaw` 0.3.0: MoE experts in system RAM, and the Qwen3.5 122B MoE.** A `pooled host` / `pooled join` device with a discrete GPU keeps some layers' routed experts in system RAM when the room's pledges can't hold a MoE model, and copies the ones each token uses into a cache on its GPU (#319, #320). `pooled host qwen3.6-35b-moe --here` now runs on a 12 GB GPU (about 38-45 tok/s on an RTX 5070), and a Mac Studio lending 25 GB with an RTX 5070 PC keeping 42 GB of experts in RAM runs the 122B at about 17 tok/s. `--ram N` sets how much RAM (default: the total less 16 GB, at most what is free less 4 GB, and 64 GB; `--ram 0` turns it off); unified memory and browsers never offload. While a device offloads, the room decodes without speculation (faster there; `POOLED_OFFLOAD_SPEC=1` turns it back on), and that device keeps GPU and RAM headroom past its cache (#323). Qwen3.5-122B-A10B (a two-file GGUF, 72.5 GB) is in `pooled pull` and `pooled host`'s list, with what it needs, and in the room page's `?dev=1` list (#316). On Windows the room node runs V8 without Maglev, which crashed Dawn (#317). The plugin bundles the same room node, so its rooms deal offload to `pooled join` devices (the gateway itself holds its layers whole), and its README says Windows is tested (#322).
- **`@pooled/cli` 0.4.0 and `@pooled/openclaw` 0.3.0: MoE experts in system RAM.** A `pooled host` / `pooled join` device with a discrete GPU keeps some layers' routed experts in system RAM when the room's pledges can't hold a MoE model, and copies the ones each token uses into a cache on its GPU (#319, #320). `pooled host qwen3.6-35b-moe --here` now runs on a 12 GB GPU (about 38-45 tok/s on an RTX 5070). `--ram N` sets how much RAM (default: the total less 16 GB, at most what is free less 4 GB, and 64 GB; `--ram 0` turns it off); unified memory and browsers never offload. While a device offloads, the room decodes without speculation (faster there; `POOLED_OFFLOAD_SPEC=1` turns it back on), and that device keeps GPU and RAM headroom past its cache (#323). On Windows the room node runs V8 without Maglev, which crashed Dawn (#317). The plugin bundles the same room node, so its rooms deal offload to `pooled join` devices (the gateway itself holds its layers whole), and its README says Windows is tested (#322).

### Changed
- **Flash prompt reading ~8x on the GB10** (#377): wide kernels (tiled GEMMs, expert-grouped MoE, GQA-shared attention) take the whole resident model from ~40 to ~330 tok/s; a 32K prompt in ~100 s instead of ~14 min. Rooms batch Flash prompts across the pipeline and in solo rooms (#381): ~190-200 tok/s on a two-node GB10 room at 2K, was ~21.
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ Models in the room today:
| Qwen 3.8 27B | Q4_0, needs ~17 GB across the room | hybrid Gated DeltaNet + attention, built-in draft layer for speculative decoding, 16K context (up to 64K) |
| Qwen 3.6 35B MoE | Q4_0, needs ~22.5 GB across the room | 256 experts, 8 active per token, so it decodes several times faster than the 27B; 32K context (up to 128K) |
| Qwen3 1.7B | Q8_0, needs ~5.6 GB across the room for 16K context, ~4 GB for 8K | small and quick, for rooms of phones and light laptops; 16K context, 8K when the room is short of memory |
| Qwen3.5 122B MoE | Q4_0, a 72.5 GB download, needs ~73 GB across the room | 256 experts, 8 active per token; in `pooled pull` / `pooled host` (CLI 0.4.0+), on the room page only with `?dev=1`. A Mac Studio plus an RTX 5070 PC keeping its experts in RAM runs it at ~17 tok/s |
| Qwen3.8 Flash (**experimental**) | Q4_0, 125B (6B active), ~99 GB on disk | 512 experts, 10 active per token; keeps most of itself on disk, so it needs far less GPU memory than its size; 64K context. See below |

Every device downloads only its own layers, from Hugging Face or from another device in the room that already has them, and keeps them cached for next time. If a device leaves, the room says so and the host deals its layers out again with one click. Speculative decoding, batched prefill and every kernel trick are checked by golden tests, so the speculative stream is the same as plain decoding.
Expand Down
Loading
Loading