From dd480f0519098780c1e69f659e4ae9497af216d5 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 16:34:30 +0100 Subject: [PATCH 001/121] docs: design for QWP ingest over ws:// in the Node.js client Adds the approved design for porting QuestDB Wire Protocol ingest to the Node.js client, using java-questdb-client 1.3.7-SNAPSHOT (8f5ed4f9) as the reference. Covers module layout, wire format, error policy, store-and-forward, config surface, testing strategy and a 13-PR stack. Records two source-provenance findings so implementers do not repeat them: docs/qwp/*.md in the parent repo were deleted by #7200 and still document a removed schema_id field, and design/qwp-nack-policy-v2.md predates the ABANDONED policy and the DATA_LOSS/PROTOCOL_VIOLATION categories. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 558 ++++++++++++++++++ 1 file changed, 558 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md new file mode 100644 index 0000000..6491d71 --- /dev/null +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -0,0 +1,558 @@ +# QWP ingest for the QuestDB Node.js client — design + +Date: 2026-08-07 +Status: approved, ready for implementation planning +Target repo: `questdb/nodejs-questdb-client` (`@questdb/nodejs-client`, currently 4.2.0) + +## 1. Goal + +Add QuestDB Wire Protocol (QWP) **ingest** over `ws://` / `wss://` to the Node.js +client, reaching functional parity with the Java client's ingest path, including +full store-and-forward. + +Today the Node client speaks only ILP — text (v1) or binary (v2/v3) rows over +HTTP or TCP, about 3.1k lines across `src/sender.ts`, `src/transport/`, and +`src/buffer/`. QWP is a different shape: a columnar, multi-table binary protocol +over a WebSocket, with an asynchronous acknowledgement stream, a +connection-scoped symbol dictionary, and a durable client-side send log. + +### 1.1 Out of scope + +Each of these is a separate future spec: + +- the query client (`QwpQueryClient`, result-batch decode, bind values); +- the `QuestDB` facade and its sender/query pooling; +- multi-host HA failover (`failover_*` keys, roles and zones); +- the UDP sender. + +## 2. Normative sources — and which ones are traps + +Pin these exactly. There are many checkouts of the QuestDB repo on any given +machine and they do not agree with each other. + +**Authoritative:** + +| Source | Pin | +|---|---| +| Java client | `java-questdb-client` @ **1.3.7-SNAPSHOT**, HEAD `8f5ed4f9`. Local checkout: `~/claude/wt/oss/wal-pending-negative/java-questdb-client`. Several other checkouts on disk are at 1.3.6-SNAPSHOT — check `core/pom.xml` before reading | +| Server-side QWP | `io.questdb.cutlass.qwp.*` in the parent `questdb` repo | +| Status-code reference | `https://questdb.com/docs/connect/wire-protocols/qwp-ingress-websocket/` — the URL `WebSocketResponse` itself cites | +| NACK policy rationale | `java-questdb-client/design/qwp-nack-policy-v2.md` — **rationale only**, see 2.1 | +| Second reference implementation | `c-questdb-client` (Rust): `src/ws/`, `src/egress/`, `src/ingress/sender/qwp_ws*` | + +**Stale — do not use:** `docs/qwp/{wire-ingress,sf-client,wire-egress,failover}.md` +in the parent `questdb` repo. These were **deleted from master** by `d1c5b03415` +*("chore(qwp): simplify the wire protocol to one version with inline schemas +(#7200)")*. Copies survive in older worktrees (`ingestion-efficiency`, +`pr-7162-review` @ `5d302a6716`) and still document a `schema_id: varint` field +that no longer exists on the wire. `QwpSchema`'s javadoc is explicit: *"The +schema section carries no mode byte and no schema id."* Following these docs +would encode a removed field. + +The code still cites `sf-client.md` and `connect-string.md` by section number in +javadoc. Those references are to the deleted documents; treat the code as truth +and the section numbers as historical breadcrumbs. + +### 2.1 Where `qwp-nack-policy-v2.md` is out of date + +The document describes itself as "implemented on `feat/nack-policy-v2`". Checked +against the shipping 1.3.7 code, it is stale in three ways: + +1. **There are four policies, not three.** `SenderError.Policy` ships + `RETRIABLE`, `RETRIABLE_OTHER`, `TERMINAL`, and **`ABANDONED`**. +2. **There are ten categories, not six.** Beyond the wire-mapped ones, + `SenderError.Category` adds `PROTOCOL_VIOLATION`, `UNKNOWN`, and + **`DATA_LOSS`**. +3. **`DICTIONARY_GAP` (0x0D) is a full category** mapped to `RETRIABLE`; the + document's policy table omits it. + +Section 7 below records the verified behaviour. Where the two disagree, the code +wins. + +## 3. Architecture + +### 3.1 Why this cannot just reuse the existing buffer/transport pair + +The current internal contracts bottom out at a single opaque blob: + +```ts +interface SenderBuffer { toBufferNew(pos?: number): Buffer | null; /* ... */ } +interface SenderTransport{ send(data: Buffer): Promise; /* ... */ } +``` + +QWP needs multi-table columnar accumulation, sealing into *frames* that carry +sequence numbers, and a response stream that arrives asynchronously and out of +band from the sends. The **row-builder half** of `SenderBuffer` +(`table`/`symbol`/`stringColumn`/`floatColumn`/…/`at`) maps 1:1 onto a columnar +accumulator and is kept exactly as-is — that is what preserves the public API. +Only the **drain half** is widened. + +### 3.2 Module layout + +All new code lives under `src/qwp/`, in four layers with no upward dependencies: + +``` +src/qwp/ + ws/ frame.ts handshake.ts socket.ts mask.ts + protocol/ constants.ts varint.ts bits.ts gorilla.ts nullBitmap.ts + columnWriter.ts tableBuffer.ts frameEncoder.ts + symbolDict.ts response.ts + sf/ engine.ts ring.ts segment.ts manifest.ts + ackWatermark.ts slotLock.ts orphanScanner.ts drainer.ts + sendLoop.ts + transport.ts + buffer.ts +``` + +- **`ws/`** — RFC 6455, hand-rolled over `net.Socket` / `tls.TLSSocket`. Ports + Java's `WebSocketFrameParser`/`WebSocketFrameWriter` and the Rust client's + HTTP response parser. QWP frames are binary-only, always `FIN=1`, never + fragmented, and use zstd at the protocol layer rather than + `permessage-deflate`, so no WebSocket library is used. +- **`protocol/`** — pure functions over `Buffer`. No I/O, no `async`. Directly + testable against golden vectors. +- **`sf/`** — store-and-forward. Ports `CursorSendEngine`, `SegmentRing`, + `SegmentManager`, `OrphanScanner`, `BackgroundDrainer`. +- **`sendLoop.ts`** — publish → wire → ACK → trim. Port of + `CursorWebSocketSendLoop`. + +### 3.3 Why hand-roll the WebSocket layer + +Both existing reference implementations hand-roll it, and the Rust client +records the reasoning in `src/ws/mod.rs`: it dropped `tungstenite` precisely +because a generic WebSocket library's feature set is all cost and no benefit for +QWP. Java goes further, building `WebSocketClient` on its own non-blocking +socket layer with per-OS epoll/kqueue/select subclasses; its only runtime +dependency is `slf4j-api`. + +For Node this means `net`/`tls` plus our own frame codec. It keeps the package's +dependency count unchanged (currently just `undici`), gives direct backpressure +via `socket.write()`'s return value and `'drain'`, allows arbitrary `X-QWP-*` +upgrade headers, and preserves the Node 20 floor. Cost is roughly 500–700 lines +plus its own tests. + +### 3.4 Runtime model + +Java's three threads collapse onto the event loop: + +| Java | Node | +|---|---| +| producer thread | the caller's own code | +| I/O send loop thread | an async task per connection | +| segment manager thread | an async task using `fs.promises` (libuv threadpool) | +| background drainer threads | async tasks, each owning its own WebSocket | + +No `worker_threads`, no native dependencies. Two Java primitives have no core +Node equivalent and are replaced: + +- **`mmap` segments** → plain files, positional writes via `fs.promises` + + `fdatasync`. Node must copy into `Buffer`s regardless, so mmap's zero-copy + reads buy much less here than in Java. +- **`flock` slot locks** → `O_EXCL` lockfile carrying pid + boot id, with a + liveness probe (see 8.3). + +### 3.5 Integration points in existing code + +All additive: + +- `src/options.ts` — add `WS`/`WSS` protocol constants and the QWP config keys + (section 9). +- `src/transport/index.ts` — `case WS: case WSS: return new QwpTransport(options)`. +- `src/buffer/index.ts` — return `QwpBuffer` for `ws`/`wss`. `protocol_version` + negotiation stays an ILP-only concern and is not consulted for QWP. +- `src/sender.ts` — public builder chain unchanged; the flush path internally + gains a publish-vs-send distinction. +- `src/index.ts` — export the new types. + +Version bump is a **minor** (4.3.0): nothing here changes existing behaviour. + +## 4. Public API + +Unchanged from the user's point of view — the protocol is a connect-string +change: + +```ts +const sender = Sender.fromConfig("ws::addr=localhost:9000;"); +await sender.table("trades") + .symbol("symbol", "ETH-USD") + .floatColumn("price", 2615.54) + .at(Date.now(), "ms"); +await sender.flush(); +``` + +New surface, mirroring Java: + +```ts +await sender.flush(); // publish; does NOT wait for ACK +const fsn = await sender.flushAndGetSequence(); // highest FSN published, or -1 +const ok = await sender.drain(30_000); // flush + await ACK watermark +sender.onError((e: SenderError) => { /* e.category, e.policy, e.fromFsn, e.toFsn */ }); +``` + +### 4.1 Flush semantics + +`flush()` resolves once the frame is **published into the store-and-forward +engine** — in RAM for memory mode, on disk for disk mode. It does *not* wait for +the server. This matches Java's `flush()` exactly. + +This differs from what `http::` means in this client today, where `flush()` +awaits the HTTP response. The difference is safe here only because +store-and-forward guarantees the bytes survive; that is why section 8 is part of +this spec rather than a follow-up. It must be called out prominently in the +README and in the migration notes. + +If the ring is at its `sf_max_total_bytes` cap, `flush()` awaits space for up to +`sf_append_deadline_millis` (default 30 s) and then throws. + +## 5. Ingest data flow + +``` +sender.table("t").symbol("s","x").doubleColumn("p",1.5).at(ts) + -> QwpBuffer routes into a per-table TableBuffer (column-wise typed arrays; + a column not set in a given row is marked null in that row's bitmap) + -> auto_flush_rows | auto_flush_bytes | auto_flush_interval, or flush() + -> frameEncoder.seal(): all dirty tables -> ONE frame, assigned an FSN + (payload optionally zstd-compressed when negotiated) + -> sf.append(frame) <-- flush() resolves here + -> sendLoop: frames after sentFsn -> WS binary frames, honouring + socket.write() backpressure and the server's X-QWP-Max-Batch-Size + -> ACK -> ackedFsn advances -> ring trims -> space frees +``` + +## 6. Wire format + +All little-endian, byte-level. + +### 6.1 Message header — 12 bytes + +``` +"QWP1" (4) | version:u8 | flags:u8 | tableCount:u16 | payloadLen:u32 +``` + +`MAGIC_MESSAGE = 0x31505751`, `VERSION = 1`. + +Flags: `DEFER_COMMIT 0x01`, `GORILLA 0x04`, `DELTA_SYMBOL_DICT 0x08`, +`ZSTD 0x10`. + +Both `GORILLA` and `DELTA_SYMBOL_DICT` are genuinely optional on the wire — +`QwpMessageCursor` branches on `isGorillaEnabled()` / `isDeltaSymbolDictEnabled()` +per message. Java's encoder always sets both, but the server accepts neither. +This is what makes the incremental stack in section 11 possible. + +### 6.2 Payload + +``` +if DELTA_SYMBOL_DICT: varint deltaStart, varint deltaCount, + deltaCount x [varint len][utf8] + +per table: [varint nameLen][utf8][varint rowCount][varint columnCount] + schema: columnCount x [varint nameLen][utf8][typeCode:u8] + columns: [null bitmap ceil(rowCount/8)] + type-specific payload +``` + +The schema section carries **no mode byte and no schema id** — columns are +always inline (post-#7200). + +### 6.3 Column payloads + +| Type | Code | Wire | +|---|---|---| +| BOOLEAN | 0x01 | bit-packed, 1 bit/value | +| BYTE / SHORT / INT / LONG | 0x02/0x03/0x04/0x05 | 1/2/4/8 B LE | +| FLOAT / DOUBLE | 0x06/0x07 | IEEE 754 4/8 B | +| SYMBOL | 0x09 | non-delta: varint dictSize, entries `[varint len][utf8]`, then varint index per non-null value. Delta mode: indices into the connection dictionary | +| TIMESTAMP / TIMESTAMP_NANOS / DATE | 0x0A/0x10/0x0B | int64; under `FLAG_GORILLA` a per-column encoding byte precedes, `0x00` = raw | +| UUID | 0x0C | 16 B LE | +| LONG256 | 0x0D | 32 B LE | +| GEOHASH | 0x0E | varint precision + `ceil(precision/8)` B per value | +| VARCHAR / BINARY | 0x0F/0x17 | `(N+1) x u32` offsets + concatenated bytes | +| DOUBLE_ARRAY / LONG_ARRAY | 0x11/0x12 | `[nDims:u8][dimLen:u32 x N][flattened LE]` | +| DECIMAL64/128/256 | 0x13/0x14/0x15 | scale (1 B, in schema) + LE unscaled 8/16/32 B | +| CHAR | 0x16 | 2 B UTF-16 code unit | +| IPv4 | 0x18 | 4 B LE, as INT | + +### 6.4 Limits (mirror server constants; enforce client-side before sending) + +`MAX_COLUMNS_PER_TABLE` 2048 · `MAX_COLUMN_NAME_LENGTH` 127 · +`MAX_TABLE_NAME_LENGTH` 127 · `MAX_SYMBOL_DICTIONARY_SIZE` 1,000,000 · +`DEFAULT_MAX_BATCH_SIZE` 16 MiB (the server advertises the real value via +`X-QWP-Max-Batch-Size`). + +The symbol cap must be enforced at registration time, before the row is +buffered, so that everything already buffered references ids the server will +accept. + +### 6.5 Handshake + +Request: `GET /write/v4` with `Sec-WebSocket-Key`, `Sec-WebSocket-Version: 13`, +`X-QWP-Client-Id: nodejs/`, `X-QWP-Max-Version: 1`, and +`Authorization: Basic|Bearer` derived from `user`/`password`/`token`. + +From the `101` response read: `X-QWP-Version`, `X-QWP-Max-Batch-Size`, +`X-QWP-Content-Encoding`, `X-QWP-Durable-Ack`, `X-QuestDB-Role`, +`X-QuestDB-Zone`. + +### 6.6 Server responses + +``` +OK : status:u8 | seq:u64 | tableCount:u16 | [nameLen:u16][name][seqTxn:i64] x n +DURABLE_ACK : status:u8 | tableCount:u16 | [nameLen:u16][name][seqTxn:i64] x n +error : status:u8 | seq:u64 | errLen:u16 | utf8 +``` + +`MAX_ERROR_MESSAGE_LENGTH` is 1024. + +## 7. Error handling + +### 7.1 Categories + +Ten, ported from `SenderError.Category`. Seven map to wire status bytes; three +are client-originated. + +| Category | Wire | Meaning | +|---|---|---| +| `SCHEMA_MISMATCH` | 0x03 | column missing, type clash, NOT NULL violated, no such table | +| `PARSE_ERROR` | 0x05 | malformed QWP payload — most likely a client bug | +| `INTERNAL_ERROR` | 0x06 | server-side fault, catch-all | +| `SECURITY_ERROR` | 0x08 | authn/authz failure | +| `WRITE_ERROR` | 0x09 | non-critical Cairo error, table not accepting writes | +| `NOT_WRITABLE` | 0x0C | node cannot serve writes; **reserved**, not currently emitted | +| `DICTIONARY_GAP` | 0x0D | delta dict began above the server's connection dict | +| `PROTOCOL_VIOLATION` | — | poison-frame detector fired | +| `DATA_LOSS` | — | durably buffered rows that will never be sent | +| `UNKNOWN` | any other | forward compatibility | + +### 7.2 Policies + +Four, from `SenderError.Policy`. There is **no drop policy** — the client never +discards data without saying so. + +| Policy | Behaviour | +|---|---| +| `RETRIABLE` | recycle the connection, replay from `ackedFsn + 1`; handler delivery is informational | +| `RETRIABLE_OTHER` | same replay, but rotate endpoints rather than back off against the same node | +| `TERMINAL` | latch; next producer call throws; bytes stay on disk | +| `ABANDONED` | the rows are gone; nothing throws and the sender keeps running; bytes preserved at `quarantinedPath` | + +### 7.3 Default mapping (`defaultPolicyFor`) + +``` +WRITE_ERROR, INTERNAL_ERROR, DICTIONARY_GAP, UNKNOWN -> RETRIABLE +NOT_WRITABLE -> RETRIABLE_OTHER +DATA_LOSS -> ABANDONED +SCHEMA_MISMATCH, PARSE_ERROR, SECURITY_ERROR, +PROTOCOL_VIOLATION, (default) -> TERMINAL +``` + +Rationale worth preserving in comments: `UNKNOWN` **fails open** so a status +byte from a newer server degrades to a retry rather than a dead sender; +`SECURITY_ERROR` mid-stream can only mean ACL denial on a *writable* node, +because read-only refusals arrive as reconnect-eligible closes, so `TERMINAL` is +correct. + +Three mappings are forced and ignore any user override: +`PROTOCOL_VIOLATION`→`TERMINAL`, `UNKNOWN`→`RETRIABLE`, `DATA_LOSS`→`ABANDONED`. + +**No policy resolver is implemented in 1.3.7** and none will be ported. The +`SenderError.Policy` javadoc describes a precedence chain +(`errorPolicyResolver` → per-category → `on_*_error` → `on_server_error` → +defaults) that does not exist in the code; `errorPolicyResolver` appears in that +javadoc and nowhere else. Building it would be inventing behaviour. + +### 7.4 WebSocket close codes carry zero policy weight + +Every close is a transport event → reconnect + replay. The guarded case, a frame +that deterministically kills the connection *without* a NACK (for example an +intermediary's frame-size limit), is caught behaviourally by the **poison-frame +detector**: + +- a server-active rejection (a `RETRIABLE` NACK, or a non-orderly close after at + least one send) counts a **strike**, keyed on the rejected frame's FSN — the + NACK-named frame, or the OK-level head-of-line frame for a close, never the + engine's trim watermark; +- `RETRIABLE_OTHER` never counts a strike (it is a verdict on the node, not the + bytes); +- orderly closes (`NORMAL_CLOSURE`, `GOING_AWAY`) never count strikes; +- `max_frame_rejections` consecutive strikes (default + `DEFAULT_MAX_HEAD_FRAME_REJECTIONS = 4`) escalate to `PROTOCOL_VIOLATION`, + which is `TERMINAL`; +- the counter resets **only** on OK-level acceptance at or beyond the suspect + frame, so re-OKs of frames *behind* it cannot launder the count. + +Below the threshold a `RETRIABLE` recycle is **paced**: the server is reachable +(it just answered), so the failed-connect backoff never engages. The recycle +parks *before* the first connect attempt, using the reconnect backoff dose — +initial, doubling per consecutive strike against the same frame, capped, plus +jitter. A NACK sequence that is making progress (a different frame each time) +resets to the initial dose. + +### 7.5 Backpressure + +The one structural difference from Java: Java spin-parks the producer thread. We +`await` a promise resolved either by ACK-driven trim or by `socket.on('drain')`, +with `sf_append_deadline_millis` as the rejection deadline. + +## 8. Store-and-forward + +Port of `CursorSendEngine` + `SegmentRing` + `SegmentManager` + `OrphanScanner` ++ `BackgroundDrainer` (~16.6k lines of Java). + +### 8.1 Model + +A chain of segments presented as one logical append-only log keyed by FSN. +Rotation when the active segment fills; ACK-driven trim of the oldest sealed +segments. Two watermarks, each single-writer: `publishedFsn` (producer) and +`ackedFsn` (I/O loop). + +**Single producer per engine.** Java states this explicitly and we inherit it: +one `Sender` is owned by one logical writer. This matches the constraint the +Node README already documents for ILP ("each worker thread needs its own Sender +instance"), so it introduces nothing new for users — but it does mean a slot +directory is owned by exactly one `Sender` at a time, which 8.3 enforces. + +Memory mode (PR 10) and disk mode (PR 11) share the ring; disk mode adds +file-backed segments, a manifest, and a persisted ack watermark. + +### 8.2 Durability + +`sf_durability` governs when `fdatasync` runs; `sf_sync_interval_millis` sets the +periodic barrier. Ordering rule: a segment's bytes must be durable before the +manifest entry that references them, so recovery never sees a manifest pointing +at bytes that are not there. + +### 8.3 Slot locking without `flock` + +Each slot directory holds a `.lock` file created `O_EXCL` containing pid and +boot id. A lock whose boot id differs from the current boot is stale by +definition. A lock with a matching boot id but a dead pid is stale after a +liveness probe. Anything else is live and the slot is skipped. This replaces +Java's `flock`, whose kernel-drops-on-exit property we lose and must emulate. + +### 8.4 Recovery and orphans + +On startup, if `drain_orphans` is enabled, scan for slot directories not held by +a live lock. Each is handed to a drainer task (bounded by +`max_background_drainers`) which opens its **own** WebSocket, replays the slot +read-only until `ackedFsn` catches the startup snapshot of `publishedFsn`, then +releases the slot. + +A drainer that fails terminally drops a `.failed` sentinel into the slot and +exits; future scans skip that slot until an operator clears it — bounded +automatic retry, then human-in-the-loop. Abandonment fires `DATA_LOSS` / +`ABANDONED` with `quarantinedPath` set. Note that a transient all-replica +failover window is **not** terminal and is retried indefinitely. + +Frames above the last commit-bearing (non-`DEFER_COMMIT`) FSN in a recovered +ring belong to a transaction whose commit frame was never published; the server +will never ACK them until a later commit covers them. Close-time drain must not +wait on ACKs that cannot arrive. + +## 9. Configuration + +Every key that exists in Java's `ConfigSchema` keeps its Java name exactly. Two +keys below (`init_buf_size`, `max_buf_size`) are pre-existing Node-client keys +with no Java counterpart; they carry over unchanged so `ws::` behaves like the +other Node protocols. Three tiers — the third is the one that is easy to get +wrong. + +**Implemented:** `addr`, `auto_flush`, `auto_flush_rows`, `auto_flush_bytes`, +`auto_flush_interval`, `user`, `password`, `token`, `tls_verify`, `tls_roots`, +`tls_roots_password`, `init_buf_size`, `max_buf_size`, `client_id`, +`connect_timeout`, `auth_timeout_ms`, `sf_dir`, `sf_max_total_bytes`, +`sf_max_segment_bytes`, `sf_durability`, `sf_append_deadline_millis`, +`sf_sync_interval_millis`, `reconnect_initial_backoff_millis`, +`reconnect_max_backoff_millis`, `reconnect_max_duration_millis`, +`max_frame_rejections`, `zstd`, `compression`, `compression_level`, +`request_durable_ack`, `transaction`, `drain_orphans`, +`max_background_drainers`, `close_flush_timeout_millis`, +`error_inbox_capacity`, `connection_listener_inbox_capacity`. + +**Accept-and-ignore, reserved** (Java: `Side.RESERVED`): `on_internal_error`, +`on_parse_error`, `on_schema_error`, `on_security_error`, `on_server_error`, +`on_write_error`. + +**Accept-and-ignore, egress/failover** — so one connect string serves both the +sender and a future query client: `target`, `failover`, `failover_backoff_initial_ms`, +`failover_backoff_max_ms`, `failover_max_attempts`, `failover_max_duration_ms`, +`query_pool_min`, `query_pool_max`, `query_close_timeout_ms`, `zone`, +`sender_pool_min`, `sender_pool_max`, `sender_id`, and the other pool keys. + +**Reject:** everything else. Unknown-key rejection is required, which is exactly +why both ignore-lists must be explicit rather than a catch-all. Java implements +this the same way and comments that "forward-compat is via the spec, not silent +ignore". + +### 9.1 zstd and the Node version floor + +`node:zlib`'s `zstdCompress` landed in **Node 22.15.0**. The client's documented +floor is Node 20 and CI runs `[20, 22, latest]`. A naive "require Node 22" rule +would still be wrong for 22.0–22.14. + +Therefore: **feature-detect**. Probe for `zstdCompressSync` at connect time. If +present, send `X-QWP-Content-Encoding: zstd` and set `FLAG_ZSTD`; if absent, +negotiate uncompressed. The floor stays at Node 20 and the CI matrix is +unchanged. + +## 10. Testing + +Four tiers, all four required. + +1. **Golden byte-vector fixtures.** A small harness in `java-questdb-client` + emits canonical frames — every column type, null bitmaps, Gorilla on and off, + delta-dictionary deltas, multi-table batches, the empty batch — to fixture + files checked into the Node repo *alongside the emitting Java SHA*. Node unit + tests assert byte-for-byte equality. This catches endianness, varint, + zig-zag, bit-packing and null-bitmap drift at the point of the mistake rather + than as a mysterious server NACK, and the recorded SHA makes drift visible. +2. **TypeScript mock QWP server.** Performs the upgrade, decodes frames, and + drives the whole error matrix on demand: each NACK status, malformed frames, + mid-frame disconnect, slow-consumer backpressure, server-initiated close, and + poison-detector escalation at 4 strikes. A real QuestDB will not produce + `INTERNAL_ERROR` or a torn frame to order. +3. **Testcontainers integration.** Extends the existing + `sender.integration.test.ts` pattern: ingest over `ws://`, then verify via SQL + that rows, types, nulls and symbols landed exactly. Requires an image with + QWP ingress enabled. +4. **Crash-recovery tests.** Spawn a child process, ingest, `SIGKILL` mid-flight, + then assert a fresh Sender recovers the orphan slot, replays from + `ackedFsn + 1`, and rows land exactly once. Plus the abandonment path: corrupt + a slot, assert it is quarantined with `quarantinedPath` set, `DATA_LOSS` / + `ABANDONED` is delivered, and the sender keeps running. + +## 11. PR stack + +Thirteen stacked PRs, each independently reviewable and green. PRs 1–8 are the +wire; 9–12 are the reliability story; PR 3 is the first point at which a user +could actually use the feature. + +| # | PR | Gate | +|---|---|---| +| 1 | `ws/`: framing, masking, handshake, net/tls socket | unit + mock server | +| 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | +| 3 | Sender wiring: `ws://` config, `QwpBuffer`/`QwpTransport`, auto-flush | **testcontainers e2e green** | +| 4 | Remaining scalar types + null bitmap | golden + e2e | +| 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | +| 6 | Delta symbol dictionary + `DICTIONARY_GAP` handling | golden + e2e | +| 7 | Gorilla timestamps + raw fallback | golden + e2e | +| 8 | defer-commit + zstd (feature-detected) | e2e both on and off | +| 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, poison detector | mock server | +| 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | +| 11 | Disk segments, manifest, ack watermark, `fdatasync` | crash tests | +| 12 | Slot locks, orphan scan, drainers, `DATA_LOSS`/`ABANDONED` | crash tests | +| 13 | Docs, examples, README support matrix, 4.3.0 release | — | + +## 12. Risks + +- **Silent wire divergence.** Mitigated by golden vectors pinned to a Java SHA. + The `schema_id` trap in section 2 is a live example of how this goes wrong. +- **Publish-semantics `flush()` before PR 10.** Between PR 3 and PR 10 there is + no retention, so an unacked frame lost to a disconnect is lost. PRs 3–9 must + document this in-tree and the feature must not be announced as + production-ready until PR 10 lands. +- **Slot-lock emulation.** `O_EXCL` + pid/boot-id is weaker than `flock`, which + the kernel releases on hard exit. A wrong liveness probe either strands data + (too conservative) or races two processes onto one slot (too aggressive). This + needs its own focused tests. +- **Event-loop stalls on large frames.** Encoding is synchronous. If frame + encode time becomes a problem, the mitigation is chunking within + `frameEncoder`, not `worker_threads`. From 4e5536cc275ecc6d3f75efbd464df0c925de1572 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 16:46:50 +0100 Subject: [PATCH 002/121] docs: correct QWP wire format after line-by-line Java re-review Re-verified the spec against java-questdb-client 1.3.7 (8f5ed4f9) rather than against prose. Seven wire-format errors found, several of which would have produced frames the server rejects or, worse, silently accepts as wrong data: - every column payload starts with a 1-byte null header; the spec omitted it - values are compacted to non-null rows (valueCount, not rowCount) everywhere - null bitmap semantics documented: bit=1 means NULL, LSB-first per byte - DECIMAL scale is written in the column payload, not the schema; the QwpConstants javadoc saying "in schema" is wrong and the spec had copied it - DATE is never Gorilla-encoded and carries no encoding byte - the timestamp encoding byte exists only when FLAG_GORILLA is set, and is still emitted (as 0x00) for columns of <= 2 values - GEOHASH precision is a single per-column varint; array shape is per value Also: SF boundary records use an alternating two-record CRC32C scheme at offsets 0 and 4096, not the "bytes before manifest entry" ordering previously asserted; CRC32C is Castagnoli, so zlib.crc32 cannot be used. X-QWP-Client-Id follows Java's / convention, not package.json. Added QWP-specific auto-flush defaults and the missing sender-level config keys. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 175 +++++++++++++++--- 1 file changed, 149 insertions(+), 26 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 6491d71..8819d04 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -247,29 +247,76 @@ if DELTA_SYMBOL_DICT: varint deltaStart, varint deltaCount, per table: [varint nameLen][utf8][varint rowCount][varint columnCount] schema: columnCount x [varint nameLen][utf8][typeCode:u8] - columns: [null bitmap ceil(rowCount/8)] + type-specific payload + columns: per column, in schema order: + [nullHeader:u8] + if nullHeader != 0: [null bitmap ceil(rowCount/8)] + type-specific payload, for valueCount values ``` The schema section carries **no mode byte and no schema id** — columns are -always inline (post-#7200). +always inline (post-#7200). The type byte is written verbatim: the client's +`QwpColumnDef.getWireTypeCode()` returns `typeCode` unchanged. Some javadoc +refers to a "null bitmap flag" in the type code; that phrasing is vestigial and +there is no such flag bit — nullability is the `nullHeader` byte below. + +### 6.2.1 Null encoding — read this carefully + +Every column payload begins with a **1-byte null header**: `0` means no nulls +and no bitmap follows; non-zero means a bitmap of `ceil(rowCount/8)` bytes +follows. Java writes `1`; a decoder must treat any non-zero value as "bitmap +present". + +Bitmap semantics (`QwpNullBitmap`): **bit `i` set means row `i` is NULL**, bit +order **LSB-first within each byte**. Row 9 is therefore byte 1, bit 1. + +Critically, **values are compacted**: the payload carries only the non-null +values. Java computes `valueCount = rowCount - nullCount` and every writer is +driven by `valueCount`, not `rowCount`. There are no placeholder slots for null +rows. This applies to fixed-width values, VARCHAR/BINARY offsets, symbol +indices, array entries — everything. Getting this wrong produces a frame whose +length is right for the wrong data. ### 6.3 Column payloads +`V` below is `valueCount` — the **non-null** row count (see 6.2.1), never +`rowCount`. + | Type | Code | Wire | |---|---|---| -| BOOLEAN | 0x01 | bit-packed, 1 bit/value | -| BYTE / SHORT / INT / LONG | 0x02/0x03/0x04/0x05 | 1/2/4/8 B LE | -| FLOAT / DOUBLE | 0x06/0x07 | IEEE 754 4/8 B | -| SYMBOL | 0x09 | non-delta: varint dictSize, entries `[varint len][utf8]`, then varint index per non-null value. Delta mode: indices into the connection dictionary | -| TIMESTAMP / TIMESTAMP_NANOS / DATE | 0x0A/0x10/0x0B | int64; under `FLAG_GORILLA` a per-column encoding byte precedes, `0x00` = raw | -| UUID | 0x0C | 16 B LE | -| LONG256 | 0x0D | 32 B LE | -| GEOHASH | 0x0E | varint precision + `ceil(precision/8)` B per value | -| VARCHAR / BINARY | 0x0F/0x17 | `(N+1) x u32` offsets + concatenated bytes | -| DOUBLE_ARRAY / LONG_ARRAY | 0x11/0x12 | `[nDims:u8][dimLen:u32 x N][flattened LE]` | -| DECIMAL64/128/256 | 0x13/0x14/0x15 | scale (1 B, in schema) + LE unscaled 8/16/32 B | -| CHAR | 0x16 | 2 B UTF-16 code unit | -| IPv4 | 0x18 | 4 B LE, as INT | +| BOOLEAN | 0x01 | bit-packed over `V` values, `ceil(V/8)` bytes, LSB-first | +| BYTE / SHORT / INT / LONG | 0x02/0x03/0x04/0x05 | `V x` 1/2/4/8 B LE | +| FLOAT / DOUBLE | 0x06/0x07 | `V x` IEEE 754 4/8 B | +| SYMBOL | 0x09 | non-delta: `varint dictSize`, `dictSize x [varint len][utf8]`, then `V x varint` index. Delta mode: **no dictionary**, just `V x varint` global id | +| TIMESTAMP / TIMESTAMP_NANOS | 0x0A/0x10 | see 6.3.1 | +| DATE | 0x0B | `V x` 8 B LE — **never Gorilla-encoded**, no encoding byte | +| UUID | 0x0C | `V x` 16 B (lo then hi, matching wire order) | +| LONG256 | 0x0D | `V x` 32 B (4 contiguous LE longs) | +| GEOHASH | 0x0E | `varint precision` **once per column**, then `V x ceil(precision/8)` B LE | +| VARCHAR / BINARY | 0x0F/0x17 | `(V+1) x u32` offsets + concatenated bytes. BINARY shares VARCHAR's layout exactly; only the byte-stream contract differs (opaque vs UTF-8) | +| DOUBLE_ARRAY / LONG_ARRAY | 0x11/0x12 | **per value**: `[nDims:u8][dimLen:u32 x nDims][prod(dims) x 8 B LE]`. Shape is per row, not per column | +| DECIMAL64/128/256 | 0x13/0x14/0x15 | `scale:u8` **once, at the start of the column payload**, then `V x` LE unscaled 8/16/32 B | +| CHAR | 0x16 | `V x` 2 B UTF-16 code unit | +| IPv4 | 0x18 | `V x` 4 B LE, as INT | + +Note on DECIMAL: `QwpConstants`' javadoc says *"[scale (1B in schema)]"*. That +is wrong — `writeDecimal64Column` emits `buffer.putByte(scale)` into the +**column payload**, and the schema section carries only name and type. Trust the +code. + +### 6.3.1 Timestamp encoding byte + +The encoding byte exists **only when `FLAG_GORILLA` is set on the message**. If +the flag is clear, timestamps are raw `V x 8 B` with no prefix byte at all. + +With the flag set, per `writeTimestampColumn`: + +- `V > 2` and the delta-of-delta fits: `0x01` (`ENCODING_GORILLA`) + bit-packed + payload; +- `V > 2` but it does not fit: `0x00` (`ENCODING_UNCOMPRESSED`) + raw `V x 8 B`; +- `V <= 2`: `0x00` + raw `V x 8 B`. + +So a Gorilla-advertising client must still emit the byte for tiny columns. DATE +is excluded from this path entirely. ### 6.4 Limits (mirror server constants; enforce client-side before sending) @@ -285,8 +332,13 @@ accept. ### 6.5 Handshake Request: `GET /write/v4` with `Sec-WebSocket-Key`, `Sec-WebSocket-Version: 13`, -`X-QWP-Client-Id: nodejs/`, `X-QWP-Max-Version: 1`, and -`Authorization: Basic|Bearer` derived from `user`/`password`/`token`. +`X-QWP-Client-Id`, `X-QWP-Max-Version: 1`, and `Authorization: Basic|Bearer` +derived from `user`/`password`/`token`. + +`X-QWP-Client-Id` follows Java's convention of `/` +— Java 1.3.7 sends the constant `"java/1.0.2"`, which is deliberately **not** the +artifact version. Node therefore sends `nodejs/` from a +dedicated constant, not `package.json`'s version. From the `101` response read: `X-QWP-Version`, `X-QWP-Max-Batch-Size`, `X-QWP-Content-Encoding`, `X-QWP-Durable-Ack`, `X-QuestDB-Role`, @@ -413,12 +465,48 @@ directory is owned by exactly one `Sender` at a time, which 8.3 enforces. Memory mode (PR 10) and disk mode (PR 11) share the ring; disk mode adds file-backed segments, a manifest, and a persisted ack watermark. -### 8.2 Durability +### 8.2 Durability — two crash-safe boundary records + +Both on-disk boundary records use the **same alternating-generation scheme**, and +it must be ported exactly: + +- `sf-manifest.bin` (`SfManifest`) — 8 KiB, magic `SFM1` (`0x314d4653`), + version 1. +- `/.ack-watermark` (`AckWatermark`) — magic `AKW1`; record layout is + `u32 magic | u32 version | i64 generation | i64 fsn | zero-fill to 59 | + u32 CRC32C of bytes [0,60)`. + +Each file holds **two independently CRC-protected 64-byte records, at offsets 0 +and 4096**. Writes alternate between them; the CRC is stored last. Recovery +selects the valid record with the greatest `generation`. The 4 KiB separation is +deliberate — it prevents a single aligned 512-byte or 4 KiB sector tear from +damaging both records. A torn update falls back to the older valid record; if +neither validates, recovery falls back to the segment-derived seed. + +Durable ACKs are cumulative (`STATUS_DURABLE_ACK fsn=N` means "everything +`<= N` is durable"), so one monotonic watermark suffices — no per-frame bitmap. +`update()` applies a monotonic clamp. + +**fsync cadence.** Ordinary ACK-only updates stay syscall-free in Java (a store +into the mmap'd inactive record). Each non-empty background disk-trim quantum +does one `msync` plus one fd `fsync`, fsyncs the slot directory **before** +unlinking, and fsyncs it **again** after the batch. Close uses the same covering +order, so the durable watermark always guards any acknowledged segment a host +crash restores. `sf_durability` governs when `fdatasync` runs; `sf_sync_interval_millis` sets the -periodic barrier. Ordering rule: a segment's bytes must be durable before the -manifest entry that references them, so recovery never sees a manifest pointing -at bytes that are not there. +periodic barrier. + +**Two consequences of the Node primitives** (expected deviations, but they change +the cost model rather than just the mechanism): + +1. Without mmap, ACK-only watermark updates are no longer free — each becomes a + positional `write()`. The trim-quantum cadence above therefore matters more in + Node than in Java, and the implementation must not write the watermark per + ACK. +2. The checksum is **CRC32C (Castagnoli)**, not CRC-32. `zlib.crc32` is + ISO-HDLC and will not interoperate. A small CRC32C implementation is required + in `sf/`, and its vectors should be part of the golden-fixture set. ### 8.3 Slot locking without `flock` @@ -465,7 +553,16 @@ wrong. `max_frame_rejections`, `zstd`, `compression`, `compression_level`, `request_durable_ack`, `transaction`, `drain_orphans`, `max_background_drainers`, `close_flush_timeout_millis`, -`error_inbox_capacity`, `connection_listener_inbox_capacity`. +`error_inbox_capacity`, `connection_listener_inbox_capacity`, +`initial_connect_retry`, `lazy_connect`, `max_batch_rows`, `max_name_len`, +`initial_credit`, `durable_ack_keepalive_interval_millis`, +`poison_min_escalation_window_millis`, +`catch_up_cap_gap_min_escalation_window_millis`. + +**Out of scope, belongs to the facade/pooling spec** — reject for now, since no +pooling exists to configure: `sender_pool_min`, `sender_pool_max`, +`buffer_pool_size`, `acquire_timeout_ms`, `idle_timeout_ms`, `max_lifetime_ms`, +`housekeeper_interval_ms`, `sender_id`. **Accept-and-ignore, reserved** (Java: `Side.RESERVED`): `on_internal_error`, `on_parse_error`, `on_schema_error`, `on_security_error`, `on_server_error`, @@ -474,15 +571,32 @@ wrong. **Accept-and-ignore, egress/failover** — so one connect string serves both the sender and a future query client: `target`, `failover`, `failover_backoff_initial_ms`, `failover_backoff_max_ms`, `failover_max_attempts`, `failover_max_duration_ms`, -`query_pool_min`, `query_pool_max`, `query_close_timeout_ms`, `zone`, -`sender_pool_min`, `sender_pool_max`, `sender_id`, and the other pool keys. +`query_pool_min`, `query_pool_max`, `query_close_timeout_ms`, `zone`. **Reject:** everything else. Unknown-key rejection is required, which is exactly why both ignore-lists must be explicit rather than a catch-all. Java implements this the same way and comments that "forward-compat is via the spec, not silent ignore". -### 9.1 zstd and the Node version floor +### 9.1 Defaults differ from ILP — do not inherit the ILP ones + +QWP's defaults come from `QwpWebSocketSender`, not from the existing Node ILP +transports (whose auto-flush row default is far higher): + +| Setting | QWP default | +|---|---| +| `auto_flush_rows` | 1,000 | +| `auto_flush_bytes` | 8 MiB | +| `auto_flush_interval` | 100 ms | +| `auth_timeout_ms` | 15,000 | +| background connect timeout | 15,000 ms | +| `max_frame_rejections` | 4 | +| `sf_append_deadline_millis` | 30,000 | + +`auto_flush_bytes` must additionally be clamped to the server-advertised +`X-QWP-Max-Batch-Size` (default 16 MiB) once the handshake completes. + +### 9.2 zstd and the Node version floor `node:zlib`'s `zstdCompress` landed in **Node 22.15.0**. The client's documented floor is Node 20 and CI runs `[20, 22, latest]`. A naive "require Node 22" rule @@ -544,7 +658,16 @@ could actually use the feature. ## 12. Risks - **Silent wire divergence.** Mitigated by golden vectors pinned to a Java SHA. - The `schema_id` trap in section 2 is a live example of how this goes wrong. + Section 2's `schema_id` trap and section 6.3's "scale in schema" javadoc error + are both live examples: in each case the prose was wrong and only the code was + right. Golden vectors are generated from the *code*, which is why they are the + primary defence rather than a nice-to-have. +- **Null compaction (6.2.1) is the single most likely correctness bug.** A + decoder that assumes `rowCount` values instead of `valueCount` produces frames + that are self-consistent in length but wrong in content, so the server may + accept them and land corrupt data rather than NACK. Golden vectors must include + a column with nulls in the first, middle, and last row, and a fully-null + column. - **Publish-semantics `flush()` before PR 10.** Between PR 3 and PR 10 there is no retention, so an unacked frame lost to a disconnect is lost. PRs 3–9 must document this in-tree and the feature must not be announced as From d46cb6f12d16f368a5e6c90cf01bcec601bca59b Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 16:55:37 +0100 Subject: [PATCH 003/121] docs: fix config-key ownership and poison escalation in QWP spec Third pass over java-questdb-client 1.3.7 (8f5ed4f9), checking the areas the first two passes had asserted rather than verified. Config: ConfigSchema assigns every key a Side, and six of the spec's assignments were wrong. max_batch_rows, initial_credit, compression, compression_level and client_id are Side.EGRESS (query client), not ingest; sender_id is Side.INGRESS, not a pooling key. There is no `zstd` key at all -- zstd is an enum value of the egress-side `compression` key, so the spec had invented a connect-string option. Pool keys are now accept-and-ignore rather than reject, matching Java, so a shared connect string does not break the sender. username/password are canonical with user/pass as aliases. Poison detector: escalation requires a strike count AND a minimum wall-clock dwell (poison_min_escalation_window_millis, default 5000), not the count alone. Java's rationale is explicit -- with pacing, four strikes can accrue in under a second behind a load balancer whose backend is briefly down, so a count-only rule escalates transients to producer-fatal terminals. The catch-up cap gap has the same two-condition shape (16 attempts + dwell). Also defines the wire primitives the spec had used ~15 times without ever specifying them: varint is unsigned LEB128 (max 10 bytes, no implicit zig-zag), zigzag is (n<<1)^(n>>63), and string is varint length + UTF-8. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 121 +++++++++++++----- 1 file changed, 86 insertions(+), 35 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 8819d04..1eaa913 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -223,6 +223,19 @@ sender.table("t").symbol("s","x").doubleColumn("p",1.5).at(ts) All little-endian, byte-level. +### 6.0 Primitives + +- **`varint`** — unsigned **LEB128**: 7 data bits per byte, high bit `0x80` set + means another byte follows. `MAX_VARINT_BYTES = 10` for a 64-bit value. It is + *unsigned*; there is no implicit zig-zag. +- **`zigzag`** — `encode(n) = (n << 1) ^ (n >> 63)`, + `decode(n) = (n >>> 1) ^ -(n & 1)`. Applied only where a codec explicitly + calls for it (Gorilla), never implicitly by `varint`. +- **`string`** — `varint` byte length followed by UTF-8 bytes. This is Java's + `putString`, and it is what every `[varint nameLen][utf8]` below expands to. + (`putUtf8` writes raw bytes with no length prefix and is not used in the frame + structure.) + ### 6.1 Message header — 12 bytes ``` @@ -425,12 +438,28 @@ detector**: - `RETRIABLE_OTHER` never counts a strike (it is a verdict on the node, not the bytes); - orderly closes (`NORMAL_CLOSURE`, `GOING_AWAY`) never count strikes; -- `max_frame_rejections` consecutive strikes (default - `DEFAULT_MAX_HEAD_FRAME_REJECTIONS = 4`) escalate to `PROTOCOL_VIOLATION`, - which is `TERMINAL`; +- escalation to `PROTOCOL_VIOLATION` (which is `TERMINAL`) requires **both** + conditions, not just the first: + 1. `max_frame_rejections` consecutive strikes + (`DEFAULT_MAX_HEAD_FRAME_REJECTIONS = 4`), **and** + 2. the suspect frame has stayed poisoned for at least + `poison_min_escalation_window_millis` + (`DEFAULT_POISON_MIN_ESCALATION_WINDOW_MILLIS = 5_000`); `0` means escalate + immediately at the strike threshold; - the counter resets **only** on OK-level acceptance at or beyond the suspect frame, so re-OKs of frames *behind* it cannot launder the count. +The dwell window is not optional polish. Java's reasoning: a strike count +measures "how many times did we look", not "how long has this been true", and +with pacing four strikes can accrue in well under a second — for example an +accepting load balancer that closes each cycle while its backend is briefly +down. Count alone would escalate that transient into a producer-fatal terminal. +Implementing only the count is a correctness bug, not a simplification. + +The orphan drainer's symbol-dict catch-up cap gap uses the same two-condition +shape: `MAX_CATCHUP_CAP_GAP_ATTEMPTS = 16` attempts **and** +`catch_up_cap_gap_min_escalation_window_millis` of dwell. + Below the threshold a `RETRIABLE` recycle is **paced**: the server is reachable (it just answered), so the failed-connect backoff never engages. The recycle parks *before* the first connect attempt, using the reconnect backoff dose — @@ -537,47 +566,57 @@ wait on ACKs that cannot arrive. ## 9. Configuration -Every key that exists in Java's `ConfigSchema` keeps its Java name exactly. Two -keys below (`init_buf_size`, `max_buf_size`) are pre-existing Node-client keys -with no Java counterpart; they carry over unchanged so `ws::` behaves like the -other Node protocols. Three tiers — the third is the one that is easy to get -wrong. - -**Implemented:** `addr`, `auto_flush`, `auto_flush_rows`, `auto_flush_bytes`, -`auto_flush_interval`, `user`, `password`, `token`, `tls_verify`, `tls_roots`, -`tls_roots_password`, `init_buf_size`, `max_buf_size`, `client_id`, -`connect_timeout`, `auth_timeout_ms`, `sf_dir`, `sf_max_total_bytes`, -`sf_max_segment_bytes`, `sf_durability`, `sf_append_deadline_millis`, -`sf_sync_interval_millis`, `reconnect_initial_backoff_millis`, -`reconnect_max_backoff_millis`, `reconnect_max_duration_millis`, -`max_frame_rejections`, `zstd`, `compression`, `compression_level`, -`request_durable_ack`, `transaction`, `drain_orphans`, -`max_background_drainers`, `close_flush_timeout_millis`, -`error_inbox_capacity`, `connection_listener_inbox_capacity`, -`initial_connect_retry`, `lazy_connect`, `max_batch_rows`, `max_name_len`, -`initial_credit`, `durable_ack_keepalive_interval_millis`, +Java's `ConfigSchema` is a single static registry in which **every key carries a +`Side`**, and the side determines who applies it. Do not infer a key's owner from +its name — several plausible-looking keys belong to the query client. Port the +registry's classification verbatim. + +**`Side.COMMON` + `Side.INGRESS` — implemented by our sender:** + +`addr` (host:port list), `username`, `password`, `token`, `tls_verify`, +`tls_roots`, `tls_roots_password`, `auth_timeout_ms`, `connect_timeout`, +`auto_flush`, `auto_flush_bytes`, `auto_flush_interval`, `auto_flush_rows`, +`close_flush_timeout_millis`, `connection_listener_inbox_capacity`, +`drain_orphans`, `durable_ack_keepalive_interval_millis`, `error_inbox_capacity`, +`initial_connect_retry`, `max_background_drainers`, `max_frame_rejections`, `poison_min_escalation_window_millis`, -`catch_up_cap_gap_min_escalation_window_millis`. +`catch_up_cap_gap_min_escalation_window_millis`, `max_name_len`, +`reconnect_initial_backoff_millis`, `reconnect_max_backoff_millis`, +`reconnect_max_duration_millis`, `request_durable_ack`, `sender_id`, +`sf_append_deadline_millis`, `sf_dir`, `sf_durability`, `sf_max_segment_bytes`, +`sf_max_total_bytes`, `sf_sync_interval_millis`, `transaction`. + +`user` and `pass` are **aliases** of `username` and `password`, registered via +`alias()`; both spellings must resolve. -**Out of scope, belongs to the facade/pooling spec** — reject for now, since no -pooling exists to configure: `sender_pool_min`, `sender_pool_max`, -`buffer_pool_size`, `acquire_timeout_ms`, `idle_timeout_ms`, `max_lifetime_ms`, -`housekeeper_interval_ms`, `sender_id`. +Plus two pre-existing Node-client keys with no Java counterpart, carried over so +`ws::` behaves like the other Node protocols: `init_buf_size`, `max_buf_size`. -**Accept-and-ignore, reserved** (Java: `Side.RESERVED`): `on_internal_error`, -`on_parse_error`, `on_schema_error`, `on_security_error`, `on_server_error`, -`on_write_error`. +**`Side.EGRESS` — accept-and-ignore.** These configure the query client, and a +shared connect string must not break the sender: `target`, `failover`, +`failover_max_attempts`, `failover_backoff_initial_ms`, `failover_backoff_max_ms`, +`failover_max_duration_ms`, `max_batch_rows`, `initial_credit`, +`buffer_pool_size`, `compression`, `compression_level`, `client_id`, `zone`. -**Accept-and-ignore, egress/failover** — so one connect string serves both the -sender and a future query client: `target`, `failover`, `failover_backoff_initial_ms`, -`failover_backoff_max_ms`, `failover_max_attempts`, `failover_max_duration_ms`, -`query_pool_min`, `query_pool_max`, `query_close_timeout_ms`, `zone`. +**`Side.POOL` — accept-and-ignore.** The facade applies these and "the two +clients ignore" them, so we ignore them too rather than reject: `sender_pool_min`, +`sender_pool_max`, `query_pool_min`, `query_pool_max`, `acquire_timeout_ms`, +`query_close_timeout_ms`, `idle_timeout_ms`, `max_lifetime_ms`, +`housekeeper_interval_ms`, `lazy_connect`. + +**`Side.RESERVED` — accept-and-ignore:** `on_internal_error`, `on_parse_error`, +`on_schema_error`, `on_security_error`, `on_server_error`, `on_write_error`. **Reject:** everything else. Unknown-key rejection is required, which is exactly -why both ignore-lists must be explicit rather than a catch-all. Java implements +why the ignore-lists must be explicit rather than a catch-all. Java implements this the same way and comments that "forward-compat is via the spec, not silent ignore". +**There is no `zstd` configuration key.** `zstd` is an enum *value* of the +egress-side `compression` key (`zstd` | `raw` | `auto`). Ingest-side zstd is +therefore purely a handshake negotiation (9.2) with no connect-string control — +do not invent a key for it. + ### 9.1 Defaults differ from ILP — do not inherit the ILP ones QWP's defaults come from `QwpWebSocketSender`, not from the existing Node ILP @@ -591,7 +630,9 @@ transports (whose auto-flush row default is far higher): | `auth_timeout_ms` | 15,000 | | background connect timeout | 15,000 ms | | `max_frame_rejections` | 4 | +| `poison_min_escalation_window_millis` | 5,000 | | `sf_append_deadline_millis` | 30,000 | +| `reconnect_max_backoff_millis` | 5,000 | `auto_flush_bytes` must additionally be clamped to the server-advertised `X-QWP-Max-Batch-Size` (default 16 MiB) once the handshake completes. @@ -672,6 +713,16 @@ could actually use the feature. no retention, so an unacked frame lost to a disconnect is lost. PRs 3–9 must document this in-tree and the feature must not be announced as production-ready until PR 10 lands. +- **Config-key ownership cannot be guessed.** `ConfigSchema` assigns every key a + `Side`, and several ingest-sounding keys (`max_batch_rows`, `initial_credit`, + `compression`, `client_id`) are `Side.EGRESS`. Port the registry as data with + its sides intact, and add a guard test asserting our classification matches + Java's key-for-key, rather than re-deriving it from key names. +- **Two-condition escalations.** Both the poison detector and the catch-up cap + gap require a strike count **and** a wall-clock dwell. Implementing the count + alone turns brief outages into producer-fatal terminals. The mock-server tests + must include a "4 strikes inside the dwell window" case that asserts *no* + escalation. - **Slot-lock emulation.** `O_EXCL` + pid/boot-id is weaker than `flock`, which the kernel releases on hard exit. A wrong liveness probe either strands data (too conservative) or races two processes onto one slot (too aggressive). This From 96ab58f6c6300706e286d620a7b0b34e009be920 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 16:59:03 +0100 Subject: [PATCH 004/121] docs: fill in QWP store-and-forward internals after fourth review pass Fourth pass, focused on section 8, which had the least line-level verification. Adds the persisted symbol dictionary (/.symbol-dict, SYD1), which the spec had omitted entirely. It is load-bearing, not an optimisation: delta SF frames carry only the symbols they introduce, so recovery and orphan adoption must re-register the whole dictionary before replay, and a surviving frame referencing a missing id is unrecoverable. Records the chunk layout, the implicit dense id numbering, the deliberate per-chunk (not per-entry) CRC32C, the write-ahead-but-not-fsynced ordering, and the rule that open() never destroys the file. Corrects slot locking: Java has TWO locks, not one. The slot .lock plus a parent-anchored logical lock under .slot-locks/, deliberately outside the slot directory so it survives a rename, which the four-step orphan adoption sequence depends on. Only the primitive is replaced in Node; the structure is not. Adds the SF01 segment file format (24-byte header, per-frame u32 crc32c + u32 payloadLen, CRC covering both), the publishedCursor publish barrier, the memoryBacked flag shared by memory and disk modes, and the four sf_durability modes (memory, periodic, flush, append), which are WebSocket-only. Splits symbol-dict persistence into its own PR; stack is now 14. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 140 +++++++++++++++--- 1 file changed, 122 insertions(+), 18 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 1eaa913..5eb8d16 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -98,7 +98,8 @@ src/qwp/ columnWriter.ts tableBuffer.ts frameEncoder.ts symbolDict.ts response.ts sf/ engine.ts ring.ts segment.ts manifest.ts - ackWatermark.ts slotLock.ts orphanScanner.ts drainer.ts + ackWatermark.ts symbolDictFile.ts crc32c.ts + slotLock.ts orphanScanner.ts drainer.ts sendLoop.ts transport.ts buffer.ts @@ -112,7 +113,8 @@ src/qwp/ - **`protocol/`** — pure functions over `Buffer`. No I/O, no `async`. Directly testable against golden vectors. - **`sf/`** — store-and-forward. Ports `CursorSendEngine`, `SegmentRing`, - `SegmentManager`, `OrphanScanner`, `BackgroundDrainer`. + `MmapSegment`, `SegmentManager`, `SfManifest`, `AckWatermark`, + `PersistedSymbolDict`, `SlotLock`, `OrphanScanner`, `BackgroundDrainer`. - **`sendLoop.ts`** — publish → wire → ACK → trim. Port of `CursorWebSocketSendLoop`. @@ -491,8 +493,78 @@ Node README already documents for ILP ("each worker thread needs its own Sender instance"), so it introduces nothing new for users — but it does mean a slot directory is owned by exactly one `Sender` at a time, which 8.3 enforces. -Memory mode (PR 10) and disk mode (PR 11) share the ring; disk mode adds -file-backed segments, a manifest, and a persisted ack watermark. +Memory mode (PR 10) and disk mode (PR 11) share the ring **and the segment +abstraction**: Java's `MmapSegment` has a `memoryBacked` flag selecting a +malloc'd buffer instead of a file mapping, with the same cursor architecture. +Port that flag rather than writing two segment types. + +**Publish barrier.** Each segment carries an `appendCursor` (producer-only) and a +`publishedCursor`. The consumer **must not read any byte at offset +`>= publishedOffset()`**. That single rule is what makes the whole thing +lock-free, and it is easy to lose in a port where `await` interleaves differently +than Java's threads. + +### 8.1.1 Segment file format (`MmapSegment`) + +``` +24-byte header: + u32 magic 'SF01' (0x31304653) | u8 version=1 | u8 flags | u16 reserved=0 + u64 baseSeq | u64 createdMicros + +then frames, each: + u32 crc32c | u32 payloadLen | payloadLen bytes +``` + +The CRC32C covers **`payloadLen` and the payload together**, not the payload +alone. `flags` bit 0 is `MANIFEST_REQUIRED_FLAG`. Segment files use the `.sfa` +extension and the mapping is sized at construction and never grows — when an +append does not fit, the caller rotates. + +Java exposes both a legacy `msync`-only flush and a checked `syncPublished()` +mapping-plus-fd barrier; only the latter is a portable power-loss barrier, so +the Node port implements the `syncPublished()` semantics (write + `fdatasync`) +and does not reproduce the legacy path. + +### 8.1.2 Persisted symbol dictionary — load-bearing, not an optimisation + +`/.symbol-dict` (`PersistedSymbolDict`) is the component most easily +missed, and omitting it makes delta-encoded recovery silently impossible. + +Delta-encoded SF frames are **not self-sufficient**: a frame carries only the +symbols it introduces. Recovering a slot after a restart, or adopting an orphan +slot, therefore requires re-registering the *whole* dictionary against the fresh +server before those frames can replay. Unlike `.ack-watermark` — a discardable +optimisation guarded by a monotonic clamp — this file is load-bearing: **a +surviving frame that references an id missing from it is unrecoverable.** + +``` +offset 0: u32 magic 'SYD1' +offset 4: u8 version = 1 +offset 5: 3 bytes reserved (zero) +offset 8: chunks, each + [entryCount: varint][entryBytes: varint][entries][crc32c: u32] + entries = [len: varint][utf8] x entryCount, occupying exactly + entryBytes bytes; the CRC-32C covers BOTH header varints and the + entry region. +``` + +Rules that must survive the port: + +- **One chunk = one append = exactly the symbols one frame introduces.** The + producer persists a frame's new symbols in a single call *before* publishing + that frame. +- **Ids are implicit.** Symbol id `i` is the `i`-th entry across all chunks; ids + are dense from 0, so no id is stored. A wrong chunk boundary silently + renumbers every later symbol. +- **CRC is per chunk, deliberately not per entry.** Every `deltaStart` is a chunk + boundary, so per-entry granularity recovers no additional prefix while adding + a checksum call per symbol on the producer path. +- **Write-ahead, but not fsynced.** Symbols are appended before the frame is + published, matching the rest of SF's page-cache (not disk) durability. This is + sufficient for a process crash — the page cache survives — but is explicitly + *not* a host-power-loss guarantee. +- **`open` never destroys it.** Only a fresh start truncates, via `openClean`, + and a failed truncation must refuse the slot outright rather than proceed. ### 8.2 Durability — two crash-safe boundary records @@ -523,8 +595,12 @@ unlinking, and fsyncs it **again** after the batch. Close uses the same covering order, so the durable watermark always guards any acknowledged segment a host crash restores. -`sf_durability` governs when `fdatasync` runs; `sf_sync_interval_millis` sets the -periodic barrier. +`sf_durability` selects one of exactly four modes — `memory`, `periodic`, +`flush`, `append` — and any other value is rejected. `sf_sync_interval_millis` +sets the periodic barrier. Both keys are **WebSocket-only**: Java throws +`"sf_durability is only supported for WebSocket transport"` if they appear with +another protocol, and the Node port must reject them the same way for `http::` +and `tcp::`. **Two consequences of the Node primitives** (expected deviations, but they change the cost model rather than just the mechanism): @@ -537,13 +613,40 @@ the cost model rather than just the mechanism): ISO-HDLC and will not interoperate. A small CRC32C implementation is required in `sf/`, and its vectors should be part of the golden-fixture set. -### 8.3 Slot locking without `flock` - -Each slot directory holds a `.lock` file created `O_EXCL` containing pid and -boot id. A lock whose boot id differs from the current boot is stale by -definition. A lock with a matching boot id but a dead pid is stale after a -liveness probe. Anything else is live and the slot is skipped. This replaces -Java's `flock`, whose kernel-drops-on-exit property we lose and must emulate. +### 8.3 Slot locking — two locks, not one + +Java's `SlotLock` provides **two distinct advisory locks**, and the second is not +optional — the orphan-adoption sequence depends on it: + +1. **Slot lock** — `acquire()` locks `/.lock` for the entire lifetime of + the owning engine. +2. **Logical lock** — `acquireLogical()` locks a sibling file under + `/.slot-locks/`, used for short-lived pathname transitions and orphan + adoption. It lives **outside** the slot directory precisely so it stays valid + if that directory is renamed. + +A drainer therefore adopts an orphan by: taking the parent-anchored logical lock +→ revalidating the scanner snapshot → taking the slot's `.lock` → releasing the +logical lock. + +Java uses real `flock` / `LockFileEx`, and writes the holder's PID to a separate +`.lock.pid` file so a failed acquisition can name the offending process. The PID +is a separate file because Windows' `LockFileEx` is a *mandatory* range lock — +while `.lock` is held, a second handle cannot read its own bytes. + +The contract being protected: two senders on one slot dir would interleave their +FSN sequences on disk and corrupt recovery. Detecting the collision at +acquisition and refusing to start is correct, because no data is on disk yet. + +**Node deviation.** Core Node exposes no `flock`, so both locks are emulated with +an `O_EXCL` lockfile containing pid + boot id: a differing boot id is stale by +definition; a matching boot id with a dead pid is stale after a liveness probe; +anything else is live and the slot is skipped. The property genuinely lost is the +kernel's automatic release on hard exit, which the boot-id/liveness probe +reconstructs. Both lock kinds and the four-step adoption order above must still +be implemented — only the primitive changes. The `.lock.pid` split is unnecessary +for us (our lockfile is advisory and readable), but the PID-in-error-message +diagnostic should be kept. ### 8.4 Recovery and orphans @@ -676,8 +779,8 @@ Four tiers, all four required. ## 11. PR stack -Thirteen stacked PRs, each independently reviewable and green. PRs 1–8 are the -wire; 9–12 are the reliability story; PR 3 is the first point at which a user +Fourteen stacked PRs, each independently reviewable and green. PRs 1–8 are the +wire; 9–13 are the reliability story; PR 3 is the first point at which a user could actually use the feature. | # | PR | Gate | @@ -692,9 +795,10 @@ could actually use the feature. | 8 | defer-commit + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, poison detector | mock server | | 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | -| 11 | Disk segments, manifest, ack watermark, `fdatasync` | crash tests | -| 12 | Slot locks, orphan scan, drainers, `DATA_LOSS`/`ABANDONED` | crash tests | -| 13 | Docs, examples, README support matrix, 4.3.0 release | — | +| 11 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | +| 12 | `.symbol-dict` persistence + delta replay after recovery | crash tests | +| 13 | Slot locks (both kinds), orphan scan, drainers, `DATA_LOSS`/`ABANDONED` | crash tests | +| 14 | Docs, examples, README support matrix, 4.3.0 release | — | ## 12. Risks From 6b092f0aa1856b329609fd458b68f68d4c4905e2 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 17:31:55 +0100 Subject: [PATCH 005/121] docs: add reconnect dict catch-up and recovery no-dedup rule to QWP spec Fifth pass, over CursorWebSocketSendLoop's reconnect/replay sequencing. Adds the symbol-dictionary catch-up, previously absent. The delta dictionary is connection-scoped on the server, so after a reconnect the fresh server's dictionary is empty while every surviving SF frame references ids from the old connection. The send loop must emit a catch-up frame re-registering from id 0 before replaying any data frame. The spec had named the outcome ("re-register and replay") without the mechanism, which is not implementable. Records the chunking rules, including that an unadvertised batch cap is NOT unbounded -- pack against 64 KiB, because the transport closes oversized frames with 1009 and a catch-up-only close is non-terminal, so an unchunked catch-up reconnects into the identical frame forever. Also the cap-gap asymmetry: a foreground sender retries forever, only an orphan drainer latches, after 16 attempts AND 300s dwell. Adds the recovery no-dedup rule for .symbol-dict replay: entries are appended unconditionally by position, because the file, the wire delta and the catch-up mirror all key on id, not string. Flags that Java's colliding case (lone UTF-16 surrogates -> '?') diverges in Node (-> U+FFFD), so those inputs must be excluded from byte-equality golden vectors. Corrects catch_up_cap_gap_min_escalation_window_millis to 300000 and adds the reconnect backoff defaults. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 80 ++++++++++++++++++- 1 file changed, 76 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 5eb8d16..e962a6c 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -459,8 +459,10 @@ down. Count alone would escalate that transient into a producer-fatal terminal. Implementing only the count is a correctness bug, not a simplification. The orphan drainer's symbol-dict catch-up cap gap uses the same two-condition -shape: `MAX_CATCHUP_CAP_GAP_ATTEMPTS = 16` attempts **and** -`catch_up_cap_gap_min_escalation_window_millis` of dwell. +shape — `MAX_CATCHUP_CAP_GAP_ATTEMPTS = 16` attempts **and** +`catch_up_cap_gap_min_escalation_window_millis` (300,000) of dwell — for the +same stated reason: a strike count measures "how many times did we look", not +"how long has this been true". See 7.5. Below the threshold a `RETRIABLE` recycle is **paced**: the server is reachable (it just answered), so the failed-connect backoff never engages. The recycle @@ -469,7 +471,50 @@ initial, doubling per consecutive strike against the same frame, capped, plus jitter. A NACK sequence that is making progress (a different frame each time) resets to the initial dose. -### 7.5 Backpressure +### 7.5 Reconnect requires a symbol-dictionary catch-up + +The delta symbol dictionary is **connection-scoped on the server**. After a +reconnect the fresh server's dictionary is empty, while every surviving frame in +the SF log references ids assigned on the old connection. Replaying data frames +directly would earn `STATUS_DICTIONARY_GAP` immediately. + +So on every reconnect, before replaying any data frame, the send loop emits a +**dictionary catch-up frame** re-registering the dictionary from id 0. This is +the mechanism behind 7.3's "`DICTIONARY_GAP` → re-register and replay"; the spec +previously named the outcome without naming the mechanism, which is not +implementable. + +Chunking rules: + +- The catch-up is packed against the server's advertised `X-QWP-Max-Batch-Size`. +- **"Not advertised" is not "unbounded."** If the server omits the header (older + build, or a derived cap that collapsed to zero), pack against + `UNCAPPED_CATCHUP_PACKING_LIMIT = 64 KiB` — deliberately well below the 128 KiB + default receive buffer. The transport still closes anything larger than the + receive buffer with WS 1009, and a catch-up-only close is deliberately + non-terminal, so an unchunked catch-up would reconnect into the identical + oversized frame forever. +- The packing limit bounds **multi-entry** packing only. A single oversized entry + is measured against a separate, more generous limit, so an entry that already + shipped inside a data frame is never reclassified as unsendable. + +**Cap gap.** If a catch-up reaches a fresh server and finds a single entry too +large for that server's cap, that is a cap-gap attempt. A homogeneous cluster +never trips it — an entry that fit its data frame under a cap always fits its +bare catch-up frame under the same cap — so it only arises in a heterogeneous or +rolling-cap cluster after failover to a smaller-cap node. + +The asymmetry matters: **a foreground sender retries forever; only an orphan +drainer may latch a terminal**, after `MAX_CATCHUP_CAP_GAP_ATTEMPTS = 16` +consecutive cap gaps *and* `catch_up_cap_gap_min_escalation_window_millis` +(default **300,000**, i.e. 5 min) of dwell. The counter increments *only* when a +node was reached and an entry was oversized. A successful catch-up ends the +episode, as does any unrelated reconnect state (connect refusal, catch-up send +failure, upgrade or role rejection) — otherwise unrelated downtime would count +toward the dwell. A cap-gap exception itself does *not* reset the episode, so +consecutive small-cap nodes still accumulate. + +### 7.6 Backpressure The one structural difference from Java: Java spin-parks the producer thread. We `await` a promise resolved either by ACK-driven trim or by `socket.on('drain')`, @@ -566,6 +611,28 @@ Rules that must survive the port: - **`open` never destroys it.** Only a fresh start truncates, via `openClean`, and a failed truncation must refuse the slot outright rather than proceed. +**Recovery replay must not de-duplicate.** Rebuilding the in-memory dictionary +from `.symbol-dict` appends every entry unconditionally at the next sequential +id (Java's `addRecoveredSymbol`, deliberately distinct from `getOrAddSymbol`). +The persisted file, the on-wire delta, and the reconnect catch-up mirror all key +on entry **position**, never on the string. If recovery collapsed two entries +that decode to the same characters, the rebuilt dictionary would be *shorter* +than the persisted entry count, desyncing the producer's delta baseline from the +catch-up mirror and silently misattributing every later symbol. For the same +reason, recovery replay is deliberately **not** capped at +`MAX_SYMBOL_DICTIONARY_SIZE` — those entries were already admitted under the cap +when first written. A reverse lookup may keep the highest id for a colliding +string; both ids encode to the same bytes, so that is harmless. + +The colliding case in Java is malformed lone UTF-16 surrogates, which its UTF-8 +encoder maps to `'?'`. **This is a live hazard in Node, not a theoretical one:** +JavaScript strings are UTF-16 and a lone surrogate (`"\uD800"`) is trivially +reachable, but Node's `Buffer.from(s, "utf8")` maps it to U+FFFD (`EF BF BD`), +not `'?'`. Node-internal consistency is preserved because everything is +position-keyed, but **Java and Node will emit different bytes for the same input +string**, so lone surrogates must be excluded from byte-equality golden vectors +and covered by a separate Node-only round-trip test. + ### 8.2 Durability — two crash-safe boundary records Both on-disk boundary records use the **same alternating-generation scheme**, and @@ -735,7 +802,12 @@ transports (whose auto-flush row default is far higher): | `max_frame_rejections` | 4 | | `poison_min_escalation_window_millis` | 5,000 | | `sf_append_deadline_millis` | 30,000 | +| `reconnect_initial_backoff_millis` | 100 | | `reconnect_max_backoff_millis` | 5,000 | +| `reconnect_max_duration_millis` | 300,000 | +| `catch_up_cap_gap_min_escalation_window_millis` | 300,000 | +| catch-up packing limit when cap unadvertised | 64 KiB | +| max catch-up cap-gap attempts (orphan drainer only) | 16 | `auto_flush_bytes` must additionally be clamped to the server-advertised `X-QWP-Max-Batch-Size` (default 16 MiB) once the handshake completes. @@ -793,7 +865,7 @@ could actually use the feature. | 6 | Delta symbol dictionary + `DICTIONARY_GAP` handling | golden + e2e | | 7 | Gorilla timestamps + raw fallback | golden + e2e | | 8 | defer-commit + zstd (feature-detected) | e2e both on and off | -| 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, poison detector | mock server | +| 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | | 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | | 11 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | | 12 | `.symbol-dict` persistence + delta replay after recovery | crash tests | From 1c489198742db680fc703f6811d07726bbcd23de Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 17:34:52 +0100 Subject: [PATCH 006/121] docs: add hot-spare provisioning and close() ordering to QWP spec Sixth pass, over SegmentManager and the close path. Adds hot-spare provisioning, absent until now. The segment manager keeps every ring supplied with a pre-created spare so segment create (open+allocate+map) and trim (unmap+unlink) never run on the producer or I/O path; rotation just swaps in the spare. Records the 1 ms poll tick, MIN_LIVE_SEGMENTS=2, and the staged trim retry. Omitting this fails no test -- it silently moves a file create onto the producer at every rotation. Records the .symbol-dict liveness-floor deadlock, which a naive port reintroduces: segment bytes are reclaimable by ACK-driven trim, so refusing to provision on them is backpressure that clears itself, but .symbol-dict bytes are lifetime-monotonic. If side-file bytes alone push a ring under sf_max_total_bytes, no ack can ever free the shortfall and the producer stalls permanently, across restarts, while the disk-full warning points at a trim that cannot help. Enforcing the cap as a directory-byte sum is the bug. Separates the two ring append sentinels, previously conflated as "backpressure": BACKPRESSURE_NO_SPARE clears itself and should wait, PAYLOAD_TOO_LARGE never clears and must fail immediately rather than burn the append deadline and report a timeout. Adds close() ordering: a pre-flight rejection of the final batch must not escape before commit/seal/drain, or it abandons rows an earlier successful flush already published; and close() must surface latched terminal errors, since a caller who only closes would otherwise never see a server rejection. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 87 ++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index e962a6c..4a7d89c 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -549,6 +549,60 @@ Port that flag rather than writing two segment types. lock-free, and it is easy to lose in a port where `await` interleaves differently than Java's threads. +### 8.1.0 Hot-spare provisioning — the producer never creates a segment + +`SegmentManager` is a background worker that keeps every registered ring +supplied with a **pre-created hot-spare segment**, and trims segments once their +frames are ACKed. The point is to keep the expensive operations — segment +creation (`open + allocate + map`) and trim (`unmap + unlink`) — **off both the +producer and the I/O path entirely**. On rotation the producer swaps in an +already-existing spare; it never waits on file creation. + +- One manager serves many rings (Java: typically every `Sender` in the JVM). In + Node this is a shared module-level async task, not one per `Sender`. +- Poll tick default **1 ms** — short enough that a producer rarely observes + `BACKPRESSURE_NO_SPARE` in steady state, long enough that an idle process does + not burn CPU. +- `MIN_LIVE_SEGMENTS = 2` (active + one spare) is the minimum working set for a + producer to advance at all. +- Trim is staged and retried with backoff (4 ms → ~1.02 s) at three distinct + points — pre-barrier, unlink, post-barrier — capped at + `MAX_TRIMS_PER_RING_PASS = 64` per ring per pass, with disk-full warnings + throttled to once per 30 s. + +Omitting hot spares does not fail a test; it just moves an `open`+`allocate` +onto the producer at every rotation. Port it. + +### 8.1.0.1 The `.symbol-dict` liveness-floor deadlock — do not reintroduce + +`sf_max_total_bytes` must **not** be enforced as a naive sum of everything in the +slot directory. Java guards this with +`livenessFloorBytes = MIN_LIVE_SEGMENTS * segmentSizeBytes`, below which the cap +check never refuses to provision, and the reasoning is worth stating in full +because a straightforward Node port reintroduces the bug exactly: + +- **Segment bytes are reclaimable.** ACK-driven trim frees them, so refusing to + provision on segment bytes is *productive* backpressure — it clears itself. +- **Side-file bytes are not.** `.symbol-dict` is lifetime-monotonic; nothing + shrinks it. + +So once side-file bytes *alone* push a ring under the cap, no ACK can ever free +the shortfall. The producer stalls **permanently, and across restarts**, while +the disk-full warning points at a trim that cannot help. Guaranteeing the minimum +working set is what turns that permanent deadlock into ordinary backpressure. + +### 8.1.0.2 Two distinct append failures + +`SegmentRing.appendOrFsn` has two sentinels and they need opposite handling: + +| Sentinel | Meaning | Handling | +|---|---|---| +| `BACKPRESSURE_NO_SPARE` (-1) | active is full, no spare ready | wait — the manager or an ACK will clear it; this is the `sf_append_deadline_millis` path | +| `PAYLOAD_TOO_LARGE` (-2) | the frame does not fit in a **fresh** segment | never clears; surface a user-facing error immediately | + +Treating `PAYLOAD_TOO_LARGE` as backpressure would burn the full append deadline +before failing, and report a timeout instead of the real cause. + ### 8.1.1 Segment file format (`MmapSegment`) ``` @@ -715,6 +769,31 @@ be implemented — only the primitive changes. The `.lock.pid` split is unnecess for us (our lockfile is advisory and readable), but the PID-in-error-message diagnostic should be kept. +### 8.3.1 `close()` ordering + +`close()` is not just teardown; two of its properties are load-bearing. + +**Ordering.** The sequence is: flush user-thread state into the engine → send the +commit message if commits are deferred → seal and swap the residual buffer → +drain on close (up to `close_flush_timeout_millis`) → tear down. A pre-flight +rejection of the final batch must **not** be allowed to escape before those +later steps run: doing so skips the commit and the drain, abandoning every row an +*earlier successful* flush already published. Java handles the rejected batch as +discardable-on-close and proceeds. + +**Terminal-error surfacing.** `close()` must report a latched terminal error. A +user who only ever calls `close()` — never `flush()` afterwards — would otherwise +never learn that the server rejected their data. Equally it must not double-report +an error instance the user already caught from an earlier call. Java snapshots the +already-surfaced error once, precisely so a terminal latched between two reads +cannot be misattributed as user-owned and silently dropped. + +Deferred commits interact here: frames above the last commit-bearing +(non-`DEFER_COMMIT`) FSN belong to a transaction whose commit was never +published, so the server will never ACK them. Close-time drain must target the +last commit boundary, not `publishedFsn`, or it waits out the full timeout on +ACKs that cannot arrive. + ### 8.4 Recovery and orphans On startup, if `drain_orphans` is enabled, scan for slot directories not held by @@ -732,7 +811,7 @@ failover window is **not** terminal and is retried indefinitely. Frames above the last commit-bearing (non-`DEFER_COMMIT`) FSN in a recovered ring belong to a transaction whose commit frame was never published; the server will never ACK them until a later commit covers them. Close-time drain must not -wait on ACKs that cannot arrive. +wait on ACKs that cannot arrive (8.3.1). ## 9. Configuration @@ -806,6 +885,7 @@ transports (whose auto-flush row default is far higher): | `reconnect_max_backoff_millis` | 5,000 | | `reconnect_max_duration_millis` | 300,000 | | `catch_up_cap_gap_min_escalation_window_millis` | 300,000 | +| segment manager poll tick | 1 ms | | catch-up packing limit when cap unadvertised | 64 KiB | | max catch-up cap-gap attempts (orphan drainer only) | 16 | @@ -899,6 +979,11 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Permanent stalls that look like disk-full.** The `.symbol-dict` liveness + floor (8.1.0.1) is the clearest example: enforce `sf_max_total_bytes` as a + naive directory-byte sum and a producer can wedge forever, across restarts, + while logging a trim warning that can never help. Crash tests must include a + slot whose side files alone approach the cap. - **Slot-lock emulation.** `O_EXCL` + pid/boot-id is weaker than `flock`, which the kernel releases on hard exit. A wrong liveness probe either strands data (too conservative) or races two processes onto one slot (too aggressive). This From a47fbd67bfd7c8674410dbe135a66a0f805cf5cd Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 17:40:05 +0100 Subject: [PATCH 007/121] docs: add cap-split flush, dual dict modes and WS control frames to QWP spec Seventh pass, over the flush split path and WebSocketClient's control frames. Adds the cap-split flush, previously absent -- section 5 claimed all dirty tables always become ONE frame. When the encoded frame exceeds serverMaxBatchSize the flush splits per table, all but the last carrying FLAG_DEFER_COMMIT. Records the two rules that make it safe: pre-flight every split frame before publishing any (otherwise an oversized later frame strands the published prefix and a later commit delivers a partial batch), and snapshot the cap exactly once per flush (the I/O side lowers it on failover to a smaller-cap node, and in Node every await inside the flush is that window). States the delivery contract explicitly: a split flush failing partway leaves a deferred prefix on the ring and the next flush re-emits the whole batch, so those rows are delivered at-least-once, duplicated. This is deliberate and within store-and-forward's contract, but the spec previously implied exactly-once by omission. Adds the two symbol-dictionary modes. Delta is not simply on/off: full-dict mode ships the whole dictionary from id 0 in every frame so replay to a fresh server can never dangle an id, and delta mode is only safe once .symbol-dict exists to reseed recovery. The mode follows from available durable state, not from a user toggle -- which is what lets PR 6 ship before PR 12. Adds WebSocket control-frame obligations: PING/PONG, the RFC 6455 5.5.1 close echo, per-frame CSPRNG masking, and the separate control-frame send buffer, so a pong is never interleaved into a partially written data frame. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 90 ++++++++++++++++++- 1 file changed, 86 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 4a7d89c..5242464 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -107,9 +107,29 @@ src/qwp/ - **`ws/`** — RFC 6455, hand-rolled over `net.Socket` / `tls.TLSSocket`. Ports Java's `WebSocketFrameParser`/`WebSocketFrameWriter` and the Rust client's - HTTP response parser. QWP frames are binary-only, always `FIN=1`, never + HTTP response parser. QWP *data* frames are binary-only, always `FIN=1`, never fragmented, and use zstd at the protocol layer rather than - `permessage-deflate`, so no WebSocket library is used. + `permessage-deflate`, so no WebSocket library is used. Control frames are still + fully implemented (see 3.3.1) — "no fragmentation" applies to data, not to the + RFC's control obligations. + +#### 3.3.1 Control frames are not optional + +- **PING → PONG**, echoing the payload. A server that pings and gets no pong + will drop the connection. +- **CLOSE → echo a CLOSE back** before closing, per RFC 6455 §5.5.1. +- Outbound PING is supported (Java exposes `sendPing`), used for liveness. +- Every client→server frame must be masked with a **fresh 4-byte key drawn + per frame from the OS CSPRNG** (`crypto.randomFillSync`), per RFC 6455 §10.3. + Do not seed a userspace PRNG once and reuse it. + +**Control frames need their own send buffer.** Java keeps a `controlFrameBuffer` +distinct from the data send buffer precisely so emitting a pong cannot clobber an +in-progress data frame. The Node analogue: a large data frame written in chunks +under backpressure must never have a control frame interleaved into the middle of +its byte stream. Either write data frames as a single `socket.write()` call, or +queue control frames behind the in-flight frame — never both writers into one +partially-written frame. - **`protocol/`** — pure functions over `Buffer`. No I/O, no `async`. Directly testable against golden vectors. - **`sf/`** — store-and-forward. Ports `CursorSendEngine`, `SegmentRing`, @@ -215,12 +235,66 @@ sender.table("t").symbol("s","x").doubleColumn("p",1.5).at(ts) -> auto_flush_rows | auto_flush_bytes | auto_flush_interval, or flush() -> frameEncoder.seal(): all dirty tables -> ONE frame, assigned an FSN (payload optionally zstd-compressed when negotiated) + ...unless the encoded frame exceeds the server's cap, in which case + it is split -- see 5.1 -> sf.append(frame) <-- flush() resolves here -> sendLoop: frames after sentFsn -> WS binary frames, honouring socket.write() backpressure and the server's X-QWP-Max-Batch-Size -> ACK -> ackedFsn advances -> ring trims -> space frees ``` +### 5.1 Splitting a flush that exceeds the server cap + +When the combined encoded frame exceeds `serverMaxBatchSize` (from +`X-QWP-Max-Batch-Size`), the flush is split so that **each non-empty table gets +its own message**. All messages except the last carry `FLAG_DEFER_COMMIT` — the +server appends without committing — and the final message omits it, triggering +the commit for the whole set. If the user already enabled deferred commit, *all* +messages carry the flag. + +Two rules make this safe, and both are easy to omit: + +**Pre-flight every split frame before publishing any of them.** If a later +table's frame is only discovered oversized mid-publish, the already-published +prefix strands on the ring and a subsequent commit delivers it as a *partial +batch*. + +**Snapshot the cap exactly once per flush.** `serverMaxBatchSize` is mutable: the +I/O side lowers it on a mid-stream failover to a smaller-cap node. Dictionary +pre-registration, the split pre-flight and the publish loop must all use one +snapshot taken at the top of the flush; if they re-read it independently, a +failover *between* the reads sizes frames against different caps and breaks the +all-or-nothing guarantee. In Node every `await` inside the flush is exactly that +failover window, so this must be a local variable, not a field read. + +**The split is deliberately not atomic across frames.** A publish failure at +frame `k > 1` (backpressure deadline, recycle timeout) leaves frames `1..k-1` on +the ring as deferred-but-uncommitted. The error propagates past the +reset-table-buffers step, so the source rows survive and the *next* flush re-emits +the whole batch; the eventual commit then commits the already-published prefix +alongside the re-sent copies. Those rows are therefore delivered +**at-least-once (duplicated), not exactly-once**. This is within +store-and-forward's at-least-once contract — a DEDUP table or a durable-ack await +absorbs the duplicate — and the symbol-dict state stays consistent on retry, +because the re-sent frames carry empty deltas. Document it; do not quietly +promise exactly-once. + +### 5.2 Two symbol-dictionary modes, not one + +The delta dictionary is not simply on or off: + +- **Full-dict mode** — every frame is self-sufficient, carrying the whole + dictionary from id 0. Recovery or orphan-drain replay to a fresh server can + therefore never dangle a symbol id. +- **Delta mode** — each frame carries only ids above the last shipped id. Used in + memory mode, and in disk mode *once the persisted `.symbol-dict` has opened*. + Safe only because a reconnect re-registers via the catch-up frame (7.5) and + recovery reseeds from the persisted file (8.1.2). + +The mode is therefore a consequence of what durable state exists, not a user +toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict +mode — which is what makes PR 6 shippable before PR 12. + ## 6. Wire format All little-endian, byte-level. @@ -939,10 +1013,10 @@ could actually use the feature. |---|---|---| | 1 | `ws/`: framing, masking, handshake, net/tls socket | unit + mock server | | 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | -| 3 | Sender wiring: `ws://` config, `QwpBuffer`/`QwpTransport`, auto-flush | **testcontainers e2e green** | +| 3 | Sender wiring: `ws://` config, `QwpBuffer`/`QwpTransport`, auto-flush, cap-split (5.1) | **testcontainers e2e green** | | 4 | Remaining scalar types + null bitmap | golden + e2e | | 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | -| 6 | Delta symbol dictionary + `DICTIONARY_GAP` handling | golden + e2e | +| 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | | 7 | Gorilla timestamps + raw fallback | golden + e2e | | 8 | defer-commit + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | @@ -979,6 +1053,14 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Delivery is at-least-once, not exactly-once.** A cap-split flush that fails + partway re-emits the whole batch on the next flush, duplicating the published + prefix (5.1). This is contractual, not a defect — but it must reach the README, + because users will otherwise assume the opposite from a durable client. +- **Mutable state re-read across an `await`.** The `serverMaxBatchSize` snapshot + rule (5.1) is the known instance; the same hazard applies anywhere the port + turns one of Java's synchronous sections into an async one. Prefer locals + captured at entry over field reads. - **Permanent stalls that look like disk-full.** The `.symbol-dict` liveness floor (8.1.0.1) is the clearest example: enforce `sf_max_total_bytes` as a naive directory-byte sum and a producer can wedge forever, across restarts, From b8a0e40129b128e5ac1b764256e9c04ca4593264 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:01:20 +0100 Subject: [PATCH 008/121] docs: add reset() watermark rollback and the three callback surfaces Eighth pass, over reset() and the listener/dispatcher surface. reset() must do more than discard buffered rows. A later flush encodes its delta as [sentMaxSymbolId+1 .. currentBatchMaxSymbolId], so a watermark left behind by the discarded batch makes even a single-row follow-up carry the whole abandoned symbol range -- hitting the very cap rejection reset() exists to clear and wedging the sender permanently. It must also reset the batch watermark to -1 and reclaim never-shipped symbol ids, which is what 1.3.7's "Return never-shipped symbol ids on reset()" does. Adds the three async callback surfaces; the spec had shown only an error callback. Records the shared delivery contract: never invoked on the I/O or producer path (a queued setImmediate dispatch in Node, since there is no dispatcher thread), bounded inbox with counted drops and a minimum capacity of 16, handler exceptions caught and logged, success connection events guaranteed per transition while failure events may coalesce, and AUTH_FAILED firing before the producer-side error is observable. Also records that the progress watermark advances only on server OK frames and that a plain OK means server-side commit, not object-store durability. Fixes non-monotonic section numbering introduced by earlier passes: 3.3.1 preceded 3.3, and the 8.1.0.x block sorted before 8.1.1. Cross-references updated. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 73 ++++++++++++++++--- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 5242464..c3c3fdf 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -110,10 +110,10 @@ src/qwp/ HTTP response parser. QWP *data* frames are binary-only, always `FIN=1`, never fragmented, and use zstd at the protocol layer rather than `permessage-deflate`, so no WebSocket library is used. Control frames are still - fully implemented (see 3.3.1) — "no fragmentation" applies to data, not to the + fully implemented (see 3.2.1) — "no fragmentation" applies to data, not to the RFC's control obligations. -#### 3.3.1 Control frames are not optional +#### 3.2.1 Control frames are not optional - **PING → PONG**, echoing the payload. A server that pings and gets no pong will drop the connection. @@ -208,10 +208,61 @@ New surface, mirroring Java: await sender.flush(); // publish; does NOT wait for ACK const fsn = await sender.flushAndGetSequence(); // highest FSN published, or -1 const ok = await sender.drain(30_000); // flush + await ACK watermark -sender.onError((e: SenderError) => { /* e.category, e.policy, e.fromFsn, e.toFsn */ }); +sender.reset(); // discard buffered rows (see 4.1) ``` -### 4.1 Flush semantics +### 4.1 `reset()` must also roll back the symbol watermark + +`reset()` discards every buffered row across **all** table buffers — but +discarding rows alone is a trap. The delta section of a later flush is encoded as +`[sentMaxSymbolId + 1 .. currentBatchMaxSymbolId]`. If the discarded batch's +watermark survives, even a single-row batch after `reset()` still carries the +whole abandoned symbol range — hitting the very cap rejection `reset()` exists to +clear, and leaving the sender **unable to flush anything at all**. + +So `reset()` must additionally set the batch symbol watermark back to `-1` (the +same value a successful flush leaves behind, and read as an empty delta) and +**reclaim the symbol ids that were allocated but never shipped**. This is what +1.3.7's "Return never-shipped symbol ids on reset()" does, and omitting it +produces a permanently wedged sender rather than a visible error. + +### 4.2 Three async callbacks, not one + +Java exposes three separate surfaces; the Node port mirrors all three: + +| Java | Fires on | +|---|---| +| `SenderErrorHandler` | rejections — carries category, policy, `fromFsn`/`toFsn`, `quarantinedPath` | +| `SenderConnectionListener` | `CONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` | +| `SenderProgressHandler` | the ACK watermark advancing | + +They share one delivery contract that must survive the port: + +- **Never invoked on the I/O or producer path.** Java uses a dedicated daemon + dispatcher thread so a slow handler cannot stall publishing or reconnect. Node + has no such thread, so callbacks must be dispatched via a queue drained on a + `setImmediate`-style tick — never called inline from the socket handler. +- **Bounded inbox, surplus dropped.** Capacity comes from `error_inbox_capacity` + and `connection_listener_inbox_capacity` (minimum **16**), and drops are + counted and readable. Without the bound, a slow user callback becomes unbounded + memory growth. +- **Handler exceptions are caught and logged**; the sender keeps running. +- **Success connection events fire on every transition; failure events may be + coalesced** under inbox pressure. `AUTH_FAILED` fires *before* the + corresponding error is observable on the producer side. + +Progress-handler semantics are narrower than they look: the watermark advances +**only on server OK frames** — a rejection never advances it — values are +strictly increasing, and one call may skip several FSNs when the server batches +frames into a single OK. Callers should compare `ackedFsn` against a target +rather than assume one call per flush. + +**A plain OK is not durability.** In non-durable-ack mode an OK acknowledges +server-side *commit*, not object-store durability. Anything gating downstream +side effects on durability must opt into `request_durable_ack`. This distinction +belongs in the README, not just here. + +### 4.3 Flush semantics `flush()` resolves once the frame is **published into the store-and-forward engine** — in RAM for memory mode, on disk for disk mode. It does *not* wait for @@ -289,7 +340,7 @@ The delta dictionary is not simply on or off: - **Delta mode** — each frame carries only ids above the last shipped id. Used in memory mode, and in disk mode *once the persisted `.symbol-dict` has opened*. Safe only because a reconnect re-registers via the catch-up frame (7.5) and - recovery reseeds from the persisted file (8.1.2). + recovery reseeds from the persisted file (8.1.5). The mode is therefore a consequence of what durable state exists, not a user toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict @@ -623,7 +674,7 @@ Port that flag rather than writing two segment types. lock-free, and it is easy to lose in a port where `await` interleaves differently than Java's threads. -### 8.1.0 Hot-spare provisioning — the producer never creates a segment +### 8.1.1 Hot-spare provisioning — the producer never creates a segment `SegmentManager` is a background worker that keeps every registered ring supplied with a **pre-created hot-spare segment**, and trims segments once their @@ -647,7 +698,7 @@ already-existing spare; it never waits on file creation. Omitting hot spares does not fail a test; it just moves an `open`+`allocate` onto the producer at every rotation. Port it. -### 8.1.0.1 The `.symbol-dict` liveness-floor deadlock — do not reintroduce +### 8.1.2 The `.symbol-dict` liveness-floor deadlock — do not reintroduce `sf_max_total_bytes` must **not** be enforced as a naive sum of everything in the slot directory. Java guards this with @@ -665,7 +716,7 @@ the shortfall. The producer stalls **permanently, and across restarts**, while the disk-full warning points at a trim that cannot help. Guaranteeing the minimum working set is what turns that permanent deadlock into ordinary backpressure. -### 8.1.0.2 Two distinct append failures +### 8.1.3 Two distinct append failures `SegmentRing.appendOrFsn` has two sentinels and they need opposite handling: @@ -677,7 +728,7 @@ working set is what turns that permanent deadlock into ordinary backpressure. Treating `PAYLOAD_TOO_LARGE` as backpressure would burn the full append deadline before failing, and report a timeout instead of the real cause. -### 8.1.1 Segment file format (`MmapSegment`) +### 8.1.4 Segment file format (`MmapSegment`) ``` 24-byte header: @@ -698,7 +749,7 @@ mapping-plus-fd barrier; only the latter is a portable power-loss barrier, so the Node port implements the `syncPublished()` semantics (write + `fdatasync`) and does not reproduce the legacy path. -### 8.1.2 Persisted symbol dictionary — load-bearing, not an optimisation +### 8.1.5 Persisted symbol dictionary — load-bearing, not an optimisation `/.symbol-dict` (`PersistedSymbolDict`) is the component most easily missed, and omitting it makes delta-encoded recovery silently impossible. @@ -1062,7 +1113,7 @@ could actually use the feature. turns one of Java's synchronous sections into an async one. Prefer locals captured at entry over field reads. - **Permanent stalls that look like disk-full.** The `.symbol-dict` liveness - floor (8.1.0.1) is the clearest example: enforce `sf_max_total_bytes` as a + floor (8.1.2) is the clearest example: enforce `sf_max_total_bytes` as a naive directory-byte sum and a producer can wedge forever, across restarts, while logging a trim warning that can never help. Crash tests must include a slot whose side files alone approach the cap. From 424691236958988ed526e42ca89d73f1cfcc467a Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:18:26 +0100 Subject: [PATCH 009/121] docs: specify the Gorilla bitstream, including the bit-reversed prefixes Ninth pass, aimed at the one wire codec the spec had left as "port QwpGorillaEncoder". Records the full delta-of-delta layout: the five DoD buckets and their bit widths, that the first two timestamps ship uncompressed with only t[2] onward entering the bitstream, the exact encoded-size formula for counts 0/1/2/>2, LSB-first packing with zero padding to a byte boundary, and the single-pass pre-validation that returns -1 when a DoD leaves signed int32 and forces the uncompressed fallback. The trap worth the pass: because packing is LSB-first, the prefix constants are bit-reversed relative to how they read. The logical prefix '10' is written as 0b01, '110' as 0b011, '1110' as 0b0111. Writing 0b10 for '10' is the obvious mistake and yields a stream that decodes into plausible-but-wrong timestamps instead of failing loudly. Java carries a javadoc table saying exactly this, which suggests it has caught people before. Confirmed the client and server bucket constants are identical. Adds the matching golden-vector requirements: every bucket boundary (0, +/-64, +/-256, +/-2048, int32 edges), a raw-fallback stream, and columns of exactly 0, 1, 2 and 3 values, since sub-3 counts take a different path. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 56 ++++++++++++++++++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index c3c3fdf..065af43 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -458,6 +458,54 @@ With the flag set, per `writeTimestampColumn`: So a Gorilla-advertising client must still emit the byte for tiny columns. DATE is excluded from this path entirely. +### 6.3.2 Gorilla bitstream + +The delta-of-delta stream, in full — this is the one codec where reading the +Java beats guessing and re-running golden vectors. + +``` +DoD = (t[n] - t[n-1]) - (t[n-1] - t[n-2]) + +DoD == 0 -> '0' 1 bit +DoD in [-64, 63] -> '10' + 7 bits 9 bits +DoD in [-256, 255] -> '110' + 9 bits 12 bits +DoD in [-2048, 2047] -> '1110' + 12 bits 16 bits +otherwise (fits int32) -> '1111' + 32 bits 36 bits +``` + +- **The first two timestamps ship uncompressed**, 8 bytes each; only `t[2]` + onward enter the bitstream. Encoded size is + `8 + 8 + ceil(totalBits / 8)` for `count > 2`, `8` for `count == 1`, `16` for + `count == 2`, `0` for `count == 0`. +- Bucket ranges are ordinary two's-complement signed ranges, so a value is + emitted as its low *n* bits. +- Bits are packed **LSB-first within each byte**, the same order as the null + bitmap (6.2.1). Trailing partial bits are zero-padded to a byte boundary. +- Pre-validate before encoding: if any DoD falls outside signed int32, Gorilla + is unusable for that column — emit `ENCODING_UNCOMPRESSED` (`0x00`) and raw + int64s instead (6.3.1). Java computes feasibility and encoded size in a single + pass and returns `-1` for "cannot encode". + +**The prefix constants are bit-reversed relative to how they read.** Because +packing is LSB-first, `writeBits(value, n)` emits bit 0 of `value` first, so the +logical prefix string must be reversed when expressed as a number: + +| Logical prefix | Value passed | Width | +|---|---|---| +| `'0'` | `0b0` | 1 | +| `'10'` | `0b01` | 2 | +| `'110'` | `0b011` | 3 | +| `'1110'` | `0b0111` | 4 | +| `'1111'` | `0b1111` | 4 | + +Writing `0b10` for `'10'` is the obvious mistake and produces a stream that +decodes into plausible-but-wrong timestamps rather than failing loudly. Java's +encoder carries a javadoc table saying exactly this, which is a good sign it has +caught people before. + +Client and server bucket constants were confirmed identical, and the server's +encoder javadoc states the two share a wire format with the decoder. + ### 6.4 Limits (mirror server constants; enforce client-side before sending) `MAX_COLUMNS_PER_TABLE` 2048 · `MAX_COLUMN_NAME_LENGTH` 127 · @@ -1068,7 +1116,7 @@ could actually use the feature. | 4 | Remaining scalar types + null bitmap | golden + e2e | | 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | | 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | -| 7 | Gorilla timestamps + raw fallback | golden + e2e | +| 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | | 8 | defer-commit + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | | 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | @@ -1090,6 +1138,12 @@ could actually use the feature. accept them and land corrupt data rather than NACK. Golden vectors must include a column with nulls in the first, middle, and last row, and a fully-null column. +- **Gorilla's prefix constants are bit-reversed** (6.3.2), and getting them wrong + yields plausible-but-wrong timestamps rather than a decode failure. Vectors + must cover every DoD bucket boundary (0, ±64, ±256, ±2048, int32 edges), a + stream that trips the raw fallback, and columns of exactly 0, 1, 2 and 3 + values — the sub-3 cases take a different path (6.3.1) and are where an + off-by-one hides. - **Publish-semantics `flush()` before PR 10.** Between PR 3 and PR 10 there is no retention, so an unacked frame lost to a disconnect is lost. PRs 3–9 must document this in-tree and the feature must not be announced as From 56e4dbf04c1650c9356a1017c7ecd0214dbe504e Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:22:07 +0100 Subject: [PATCH 010/121] docs: correct the Node-side integration scope in the QWP spec Tenth pass, over section 3.5 -- claims made in the first pass about the existing Node code and never re-verified since. Checked against current main; "all additive" understated it. options.ts branches on protocol in four places, not one: the protocol token switch, parseProtocolVersion, parseAddress's port defaulting, and the doc comment. Two of them carry their own copy of the "accepted protocols" error string, which sender.config.test.ts almost certainly asserts verbatim. ws/wss default to port 9000, not 9009. Records a silent-failure hazard: parseProtocolVersion's default arm assigns PROTOCOL_VERSION_V1 to any non-HTTP protocol, and createBuffer switches on protocol_version alone, so a ws:: sender would receive SenderBufferV1 -- the ILP text buffer -- and emit ILP with no error raised. No protocol_version value denotes QWP, so createBuffer must branch on protocol before reaching that switch. Added as a risk with a required PR 3 test. Notes that resolveAuto needs no guard only by accident (it returns early on a non-auto version before building a ws://.../settings URL), so that deserves a regression test. Corrects 9.1: auto_flush_bytes is not a different default, it does not exist in the Node client at all, and auto_flush_interval is a hardcoded 1s module constant with no per-transport hook. Both are new functionality in PR 3. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 73 +++++++++++++++---- 1 file changed, 59 insertions(+), 14 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 065af43..81404ed 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -175,18 +175,49 @@ Node equivalent and are replaced: ### 3.5 Integration points in existing code -All additive: - -- `src/options.ts` — add `WS`/`WSS` protocol constants and the QWP config keys - (section 9). -- `src/transport/index.ts` — `case WS: case WSS: return new QwpTransport(options)`. -- `src/buffer/index.ts` — return `QwpBuffer` for `ws`/`wss`. `protocol_version` - negotiation stays an ILP-only concern and is not consulted for QWP. -- `src/sender.ts` — public builder chain unchanged; the flush path internally - gains a publish-vs-send distinction. -- `src/index.ts` — export the new types. - -Version bump is a **minor** (4.3.0): nothing here changes existing behaviour. +Backwards-compatible, but **not** merely "add a case" — verified against the +current `main`, the existing code branches on protocol in four places and on +`protocol_version` in a fifth. + +`src/options.ts` — four edit sites: + +1. The protocol token switch (`case HTTP: case HTTPS: case TCP: case TCPS:`) and + its error string enumerating `'http', 'https', 'tcp', 'tcps'`. +2. `parseProtocolVersion` — see the hazard below. +3. `parseAddress`'s port-defaulting switch and *its own* copy of that same error + string. `ws`/`wss` default to **9000**, the HTTP port, not 9009. +4. The `SenderOptions` doc comment listing accepted protocols. + +Both error strings are very likely asserted verbatim in `sender.config.test.ts`, +so PR 3 touches those tests. + +**Hazard — `createBuffer` must branch on protocol before `protocol_version`.** +`parseProtocolVersion` has a `default:` arm assigning `PROTOCOL_VERSION_V1` to +any protocol that is not HTTP/HTTPS. A `ws::` sender therefore reaches +`createBuffer` carrying `protocol_version = 1`, and `createBuffer` switches on +`protocol_version` alone — so it returns `SenderBufferV1`, the **ILP text +buffer**, for a QWP sender. Silently: no error, wrong bytes on the wire. No +`protocol_version` value means QWP, so adding a case to that switch is not an +option. `createBuffer` must consult `options.protocol` first and return +`QwpBuffer` before reaching the version switch, and `parseProtocolVersion` must +leave `ws`/`wss` unset rather than stamping V1. + +`SenderOptions.resolveAuto` happens to need no guard: it calls +`parseProtocolVersion`, sees a non-`auto` value and returns early, so it never +builds a `ws://host:port/settings` URL. That is accidental rather than designed, +so it warrants a regression test. + +`src/transport/index.ts` — `case WS: case WSS: return new QwpTransport(options)`. + +`src/sender.ts` — the public builder chain is unchanged, but two auto-flush gaps +must close (see 9.1): the client has **no `auto_flush_bytes` option at all**, and +`DEFAULT_AUTO_FLUSH_INTERVAL` is a hardcoded 1 s module constant with no +per-transport hook — unlike rows, which already delegate to +`transport.getDefaultAutoFlushRows()`. + +`src/index.ts` — export the new types. + +Version bump is a **minor** (4.3.0): no existing behaviour changes. ## 4. Public API @@ -1042,7 +1073,16 @@ do not invent a key for it. ### 9.1 Defaults differ from ILP — do not inherit the ILP ones QWP's defaults come from `QwpWebSocketSender`, not from the existing Node ILP -transports (whose auto-flush row default is far higher): +transports (whose auto-flush row default is far higher). Two rows below are +**not** merely different defaults — they are functionality the Node client does +not have yet: + +- **`auto_flush_bytes` does not exist** in the Node client at all. Byte-based + auto-flush must be added to `Sender`, not just defaulted. +- **`auto_flush_interval` has no per-transport hook.** It is a hardcoded 1 s + module constant in `sender.ts`; rows already delegate to + `transport.getDefaultAutoFlushRows()`, and the interval needs the same + treatment to reach QWP's 100 ms. | Setting | QWP default | |---|---| @@ -1112,7 +1152,7 @@ could actually use the feature. |---|---|---| | 1 | `ws/`: framing, masking, handshake, net/tls socket | unit + mock server | | 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | -| 3 | Sender wiring: `ws://` config, `QwpBuffer`/`QwpTransport`, auto-flush, cap-split (5.1) | **testcontainers e2e green** | +| 3 | Sender wiring: `ws://` config (4 sites, 3.5), `QwpBuffer`/`QwpTransport`, byte + interval auto-flush, cap-split (5.1) | **testcontainers e2e green** | | 4 | Remaining scalar types + null bitmap | golden + e2e | | 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | | 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | @@ -1158,6 +1198,11 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **A `ws::` sender silently falling back to ILP v1.** `parseProtocolVersion` + stamps `PROTOCOL_VERSION_V1` on any non-HTTP protocol and `createBuffer` + switches on that value alone (3.5). Get the ordering wrong and QWP emits ILP + text with no error at all. PR 3 needs an explicit test that a `ws::` sender + produces a `QwpBuffer`. - **Delivery is at-least-once, not exactly-once.** A cap-split flush that fails partway re-emits the whole batch on the next flush, duplicating the published prefix (5.1). This is contractual, not a defect — but it must reach the README, From bf84b5757e755f2556042f32594d06083c17d9f8 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:24:14 +0100 Subject: [PATCH 011/121] docs: resolve single-endpoint scope contradiction and stale testing claims Eleventh pass, reading the document as a document rather than checking it against sources. Ten passes of accreted patching had left three internal contradictions that source-comparison could not surface. Section 1.1 excludes multi-host failover, yet five later sections lean on endpoint rotation: RETRIABLE_OTHER's whole definition, two connection event kinds, the cap-snapshot rationale, the catch-up cap gap, and addr described as a host:port list. Adds 1.2 stating the stack targets a single endpoint and giving each behaviour's single-host meaning, keeping the enum and event shapes intact so the later HA spec stays additive. Cross-references added at each site. Fixes two stale claims in section 10 that later passes had invalidated. Tier 2 still described poison escalation as firing at 4 strikes, contradicting 7.4, where escalation needs the strike count AND the dwell window; it now requires both conditions be exercised, including a case that accrues 4 strikes inside the window and asserts escalation does not fire. Tier 4 still asserted rows land "exactly once", contradicting 5.1, where replay and cap-split retry legitimately duplicate; the assertion is now every row present, with duplicates explicitly not a failure. Also adds the liveness-floor case to the crash-recovery tier. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 43 +++++++++++++++---- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 81404ed..4f4cc03 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -25,6 +25,23 @@ Each of these is a separate future spec: - multi-host HA failover (`failover_*` keys, roles and zones); - the UDP sender. +### 1.2 Single endpoint — and what that degrades + +Because multi-host failover is out of scope, this stack connects to **one** +endpoint and reconnects to that same endpoint. Several behaviours ported from +Java are phrased in terms of endpoint rotation; their *shapes* are kept intact so +the later HA spec is additive rather than a rewrite, but their single-host +meaning must be stated or an implementer will either build HA by accident or +silently drop them. + +| Behaviour | Single-endpoint meaning | +|---|---| +| `RETRIABLE_OTHER` (7.2) | Keep the distinct policy and category, but with nothing to rotate to it behaves as `RETRIABLE` with the zero-progress pacer. Do not collapse the enum. | +| `FAILED_OVER`, `ALL_ENDPOINTS_UNREACHABLE` (4.2) | Defined but never emitted. `ENDPOINT_ATTEMPT_FAILED`, `CONNECTED`, `RECONNECTED`, `AUTH_FAILED` all still fire. | +| Cap changing mid-stream (5.1) | Still reachable — a reconnect to a restarted or upgraded server can advertise a different `X-QWP-Max-Batch-Size`. The snapshot-once rule stands on its own merits. | +| Catch-up cap gap (7.5) | Effectively unreachable single-host, but retained: it costs one counter and becomes live the moment HA lands. | +| `addr` | Parsed as a single `host:port`. Accept a comma-separated list syntactically if Java does, but use only the first entry, and say so rather than failing obscurely. | + ## 2. Normative sources — and which ones are traps Pin these exactly. There are many checkouts of the QuestDB repo on any given @@ -264,7 +281,7 @@ Java exposes three separate surfaces; the Node port mirrors all three: | Java | Fires on | |---|---| | `SenderErrorHandler` | rejections — carries category, policy, `fromFsn`/`toFsn`, `quarantinedPath` | -| `SenderConnectionListener` | `CONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` | +| `SenderConnectionListener` | `CONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` — two of these are never emitted single-endpoint (1.2) | | `SenderProgressHandler` | the ACK watermark advancing | They share one delivery contract that must survive the port: @@ -601,7 +618,7 @@ discards data without saying so. | Policy | Behaviour | |---|---| | `RETRIABLE` | recycle the connection, replay from `ackedFsn + 1`; handler delivery is informational | -| `RETRIABLE_OTHER` | same replay, but rotate endpoints rather than back off against the same node | +| `RETRIABLE_OTHER` | same replay, but rotate endpoints rather than back off against the same node (single-endpoint behaviour: 1.2) | | `TERMINAL` | latch; next producer call throws; bytes stay on disk | | `ABANDONED` | the rows are gone; nothing throws and the sender keeps running; bytes preserved at `quarantinedPath` | @@ -1026,7 +1043,8 @@ registry's classification verbatim. **`Side.COMMON` + `Side.INGRESS` — implemented by our sender:** -`addr` (host:port list), `username`, `password`, `token`, `tls_verify`, +`addr` (single `host:port` in this stack — see 1.2), `username`, `password`, +`token`, `tls_verify`, `tls_roots`, `tls_roots_password`, `auth_timeout_ms`, `connect_timeout`, `auto_flush`, `auto_flush_bytes`, `auto_flush_interval`, `auto_flush_rows`, `close_flush_timeout_millis`, `connection_listener_inbox_capacity`, @@ -1130,17 +1148,24 @@ Four tiers, all four required. 2. **TypeScript mock QWP server.** Performs the upgrade, decodes frames, and drives the whole error matrix on demand: each NACK status, malformed frames, mid-frame disconnect, slow-consumer backpressure, server-initiated close, and - poison-detector escalation at 4 strikes. A real QuestDB will not produce - `INTERNAL_ERROR` or a torn frame to order. + poison-detector escalation. A real QuestDB will not produce `INTERNAL_ERROR` + or a torn frame to order. Escalation needs **both** its conditions exercised + (7.4): a case that accrues 4 strikes *inside* the dwell window and asserts + that escalation does **not** fire, alongside one that crosses both. 3. **Testcontainers integration.** Extends the existing `sender.integration.test.ts` pattern: ingest over `ws://`, then verify via SQL that rows, types, nulls and symbols landed exactly. Requires an image with QWP ingress enabled. 4. **Crash-recovery tests.** Spawn a child process, ingest, `SIGKILL` mid-flight, - then assert a fresh Sender recovers the orphan slot, replays from - `ackedFsn + 1`, and rows land exactly once. Plus the abandonment path: corrupt - a slot, assert it is quarantined with `quarantinedPath` set, `DATA_LOSS` / - `ABANDONED` is delivered, and the sender keeps running. + then assert a fresh Sender recovers the orphan slot and replays from + `ackedFsn + 1` with **no row lost**. Assert *at-least-once*, not + exactly-once — replay and cap-split retry both legitimately duplicate (5.1), + so the assertion is "every row present", not "every row once", and a + duplicate must not fail the test. Plus the abandonment path: corrupt a slot, + assert it is quarantined with `quarantinedPath` set, `DATA_LOSS` / + `ABANDONED` is delivered, and the sender keeps running. Plus the liveness + floor (8.1.2): a slot whose side files alone approach `sf_max_total_bytes` + must still accept writes. ## 11. PR stack From a2e69e6ea0fbbc72c22ae8a4f995a51f8ac41f60 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:27:45 +0100 Subject: [PATCH 012/121] docs: specify upgrade-failure classification and TLS trust-store mapping Twelfth pass, read as an implementer walking PR 1 and asking what the spec could not answer. Two gaps, both PR-1 blocking. Section 6.5 described the handshake request and response headers but said nothing about a non-101 response. Java classifies three ways and conflating them inverts the retry behaviour: 421 carrying X-QuestDB-Role is a role reject retried indefinitely (the connect-time half of the read-only case), 401/403 is a terminal credential failure that emits AUTH_FAILED before the producer-side error, and everything else including 404 falls through unclassified. Treating 401 as retriable spins forever; treating 421 as terminal kills a sender during an ordinary failover window. Records that tls_roots does not port. Java takes a JVM keystore path plus password; Node's tls.connect takes PEM via ca or PKCS#12 via pfx, and cannot read JKS at all. Specifies accepting PEM and PKCS#12, detecting JKS by its 0xFEEDFEED magic and failing with a message naming the conversion rather than a parse error. Also carries over Java's rule that a custom trust store may not be combined with disabled validation. Both added to the risks, and PR 1's row now names them. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 47 ++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 4f4cc03..9ff22e9 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -580,6 +580,42 @@ From the `101` response read: `X-QWP-Version`, `X-QWP-Max-Batch-Size`, `X-QWP-Content-Encoding`, `X-QWP-Durable-Ack`, `X-QuestDB-Role`, `X-QuestDB-Zone`. +### 6.5.1 Upgrade failure classification + +A non-101 response is not one error. Java classifies three ways, and getting +this wrong inverts the retry behaviour — a credential failure retried forever, a +transient role reject treated as fatal: + +| Response | Meaning | Handling | +|---|---|---| +| `421` **with** an `X-QuestDB-Role` header | Role reject: this node cannot accept writes (read-only replica, demoting primary) | **Retried indefinitely** — never terminal. This is the connect-time half of the read-only case whose mid-stream half arrives as a reconnect-eligible close (7.4). | +| `401` / `403` | Credential failure | Terminal. Emits `AUTH_FAILED` on the connection listener *before* the producer-side error becomes observable (4.2). | +| anything else, **including `404`** | Generic upgrade failure — `404` specifically means a per-endpoint path mismatch | Surfaced as-is; not specially classified. | + +The `421` rule is why §8.4 can say a transient all-replica window is never +quarantined on a wall-clock budget: connect-time role rejects retry forever by +design. + +### 6.5.2 TLS — `tls_roots` does not port directly + +`tls_verify` maps cleanly: `on` → default verification, `unsafe_off` → +`rejectUnauthorized: false`. Java additionally enforces that **a custom trust +store may not be combined with disabled validation** (its constructor throws); +reproduce that validation rather than silently ignoring one of the two. + +`tls_roots` / `tls_roots_password` do **not** port cleanly. Java takes a +`trustStorePath` plus a `char[]` password — a JVM keystore. Node's `tls.connect` +accepts `ca` as PEM, or `pfx` + `passphrase` for PKCS#12. **JKS is not readable +by Node at all**, and no amount of option-mapping changes that. + +Decision for this stack: accept **PEM** for `tls_roots` (mapped to `ca`, password +ignored, and warn if one is supplied since PEM roots are not encrypted), and +accept **PKCS#12** (`.p12`/`.pfx`, mapped to `pfx` + `passphrase`). Detect a JKS +file by its magic bytes (`0xFEEDFEED`) and fail with an explicit "JKS keystores +are not supported by the Node client; convert to PKCS#12 or PEM" — not a parse +error. A connect string that works against Java may therefore fail here, so this +belongs in the README's compatibility notes, not only in this spec. + ### 6.6 Server responses ``` @@ -1175,7 +1211,7 @@ could actually use the feature. | # | PR | Gate | |---|---|---| -| 1 | `ws/`: framing, masking, handshake, net/tls socket | unit + mock server | +| 1 | `ws/`: framing, masking, handshake, upgrade-failure classification (6.5.1), TLS mapping (6.5.2), net/tls socket | unit + mock server | | 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | | 3 | Sender wiring: `ws://` config (4 sites, 3.5), `QwpBuffer`/`QwpTransport`, byte + interval auto-flush, cap-split (5.1) | **testcontainers e2e green** | | 4 | Remaining scalar types + null bitmap | golden + e2e | @@ -1223,6 +1259,15 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Inverted upgrade-failure retry.** Treating `401`/`403` as retriable spins + forever against a server that will never accept the credentials; treating + `421` as terminal kills a sender during an ordinary failover window (6.5.1). + The two are easy to conflate because both are "the server refused the + upgrade". Mock-server tests must cover `421`-with-role, `401`, `403`, `404`. +- **A connect string that works on Java failing on Node.** `tls_roots` with a + JKS keystore is unsupportable in Node (6.5.2). Fail with an explicit message + naming the conversion, and document it — a silent parse failure here looks + like a client bug. - **A `ws::` sender silently falling back to ILP v1.** `parseProtocolVersion` stamps `PROTOCOL_VERSION_V1` on any non-HTTP protocol and `createBuffer` switches on that value alone (3.5). Get the ordering wrong and QWP emits ILP From 9ad6d6936c2ba8932dbab5500391656a5a99c9e6 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:30:20 +0100 Subject: [PATCH 013/121] docs: add the commit frame, row rollback, and missing wire limits Thirteenth pass, walking PR 2 and PR 3 as an implementer from the ground up. Adds the commit frame (5.1.1), an entire wire message the spec had never described: tableCount 0, no rows, FLAG_DEFER_COMMIT cleared, and no symbols. The empty delta must be built by construction, pinning both bounds to the baseline, because the commit path does not write-ahead-persist the dictionary -- shipping a symbol there puts an id on the wire that a recovered slot cannot rebuild, silently misattributing reused ids after a crash. Deriving the bound from batch state instead is a bug Java already fixed: the batch watermark is not reliably reset after an empty flush or a cancelled row, and a commit reaching that window re-shipped the entire dictionary in a frame no chunker covers. Adds the row-rollback invariant (4.1.1), which has no ILP analogue. In a row-oriented buffer a half-written row is trailing bytes; in a columnar one a setter that throws mid-row leaves columns at unequal lengths, so every later frame is malformed while still looking structurally valid. Java wraps every column setter in rollbackRow; the port must too. Notes that Node's Sender has no cancelRow, so that parity is optional while the rollback is not. Adds the wire limits the spec was missing: DEFAULT_MAX_ROWS_PER_TABLE 1,000,000 and DEFAULT_MAX_TABLES_PER_CONNECTION 10,000, plus the u16 structural ceiling on tableCount. Clarifies that payloadLen excludes the header and that one QWP message is exactly one WebSocket binary frame. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 81 ++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 9ff22e9..582b4b4 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -274,6 +274,31 @@ same value a successful flush leaves behind, and read as an empty delta) and 1.3.7's "Return never-shipped symbol ids on reset()" does, and omitting it produces a permanently wedged sender rather than a visible error. +### 4.1.1 A throwing column setter must roll back the row + +This is a columnar-specific invariant with no ILP analogue, and it is easy to +miss because the row-oriented client never needed it. + +In ILP a half-written row is just trailing bytes in one buffer — truncate and +continue. In QWP each column has its own value array, so a setter that throws +midway through a row leaves the table buffer **desynchronised**: some columns +hold `N` values, the ones already set hold `N+1`. Every later frame from that +buffer is then malformed, and the null-bitmap/`valueCount` accounting (6.2.1) +silently attributes values to the wrong rows. + +Java wraps every column setter in `catch (RuntimeException | Error e) { +rollbackRow(); throw e; }`. The Node port must do the same: any throw from a +column setter — validation failure, cap rejection, type error — rolls the +in-progress row back to the last committed row boundary across **all** columns +before propagating. + +Java also exposes `cancelRow()` for explicit abandonment. The Node `Sender` has +no such method today; adding it is optional for this stack, but the internal +rollback it shares is **not** optional. Note the interaction in 5.1.1: a +cancelled or rolled-back row can leave a symbol registered in the dictionary, +which is harmless provided the commit frame pins both delta bounds to the +baseline. + ### 4.2 Three async callbacks, not one Java exposes three separate surfaces; the Node port mirrors all three: @@ -378,6 +403,39 @@ absorbs the duplicate — and the symbol-dict state stays consistent on retry, because the re-sent frames carry empty deltas. Document it; do not quietly promise exactly-once. +### 5.1.1 The commit frame + +Deferred commits need a message that commits without carrying data. It is a +normal QWP frame with `tableCount = 0`, no rows, `FLAG_DEFER_COMMIT` **cleared** +— and, critically, **no symbols**. + +The empty delta must be produced *by construction*, by passing the current +baseline as **both** bounds so the range is `[baseline+1 .. baseline]`. This is +the only shape that is unconditionally correct in both dictionary modes (5.2): + +- **Delta mode** — the commit path does *not* write-ahead-persist the dictionary + (8.1.5). Shipping a symbol here would put an id on the wire that a recovered + slot cannot rebuild from `.symbol-dict`, diverging the producer's dictionary + from the surviving frames and **silently misattributing reused ids after a + crash**. +- **Full-dict mode** — the baseline is `-1`, so the frame carries `deltaStart 0` + with a zero count. Nothing needs registering; the group's data frames already + did it. + +Deriving the upper bound from the current batch's max symbol id instead is a +**bug Java already fixed**, and the failure mode is worth knowing because it is +invisible in the common case. That value is not reliably reset: `flushPendingRows` +returns early without clearing it when there are no pending rows or every table +is empty, and `cancelRow` leaves a registered symbol's id behind. A commit +reaching that window re-shipped the **entire dictionary from id 0**, in a frame +that no cap check and no chunker covers — reintroducing the oversized-frame wall +on the single path that bypasses the splitter (5.1). + +Any symbol leaked by a cancelled row is picked up by the next real flush, whose +write-ahead persist resumes from the persisted dictionary's size. The commit +frame also sets the last-commit-boundary FSN, which close-time drain depends on +(8.3.1). + ### 5.2 Two symbol-dictionary modes, not one The delta dictionary is not simply on or off: @@ -419,6 +477,11 @@ All little-endian, byte-level. `MAGIC_MESSAGE = 0x31505751`, `VERSION = 1`. +`payloadLen` counts the payload **only**; total message length is +`HEADER_SIZE + payloadLen`. One QWP message is carried as exactly one WebSocket +binary frame — the QWP header is the first byte of the WS payload, never split +across frames. + Flags: `DEFER_COMMIT 0x01`, `GORILLA 0x04`, `DELTA_SYMBOL_DICT 0x08`, `ZSTD 0x10`. @@ -558,9 +621,14 @@ encoder javadoc states the two share a wire format with the decoder. `MAX_COLUMNS_PER_TABLE` 2048 · `MAX_COLUMN_NAME_LENGTH` 127 · `MAX_TABLE_NAME_LENGTH` 127 · `MAX_SYMBOL_DICTIONARY_SIZE` 1,000,000 · +`DEFAULT_MAX_ROWS_PER_TABLE` 1,000,000 · +`DEFAULT_MAX_TABLES_PER_CONNECTION` 10,000 · `DEFAULT_MAX_BATCH_SIZE` 16 MiB (the server advertises the real value via `X-QWP-Max-Batch-Size`). +`tableCount` is a `u16`, so 65,535 is a hard structural ceiling independent of +`DEFAULT_MAX_TABLES_PER_CONNECTION`. + The symbol cap must be enforced at registration time, before the row is buffered, so that everything already buffered references ids the server will accept. @@ -1218,7 +1286,7 @@ could actually use the feature. | 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | | 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | | 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | -| 8 | defer-commit + zstd (feature-detected) | e2e both on and off | +| 8 | defer-commit + commit frame (5.1.1) + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | | 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | | 11 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | @@ -1259,6 +1327,17 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **A throwing setter desynchronising the columns.** Unequal per-column lengths + (4.1.1) corrupt every subsequent frame from that table buffer while each frame + still looks structurally valid. Tests must throw from a setter mid-row — first + column, middle, last — and assert the next flush is byte-identical to one where + the row was never started. +- **The commit frame re-shipping the whole dictionary.** Deriving its symbol + bound from batch state rather than pinning both bounds to the baseline (5.1.1) + produces a correct-looking frame in the common case and an unsplittable + oversized one after a cancelled row or an empty flush. Java hit this; the + golden vectors must include a commit frame emitted after `cancelRow` and after + an empty flush. - **Inverted upgrade-failure retry.** Treating `401`/`403` as retriable spins forever against a server that will never accept the credentials; treating `421` as terminal kills a sender during an ordinary failover window (6.5.1). From 9cba6dcb651e39e199b6690aff156439a8ed159a Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:34:18 +0100 Subject: [PATCH 014/121] docs: cover PR 4/5 column rules and the remaining ingest classes Fourteenth pass, cross-referencing the Java client class by class against the spec and filling the PR 4/5 surface, which had no coverage at all. Adds 6.5.3, the per-column accumulation rules from QwpTableBuffer: type locked on first sight, duplicate column within a row silently first-value-wins, row completion back-filling nulls so columns stay equal length, and a per-row size guard distinct from the per-frame split. Per type: geohash precision locked 1-60, BINARY nulls only via the bitmap, jagged arrays rejected, 2 GiB string data cap per batch, and no mixing global symbol ids with a local dictionary in one column. The decimal rule is the trap: scale is locked on the first value and later values are automatically RESCALED to it, throwing only on precision loss or capacity overflow. Mis-porting it as lock-and-reject rejects data Java accepts, order-dependently. Adds two post-101 upgrade failures 6.5.1 did not cover, with opposite handling. An unsupported X-QWP-Version is transient at every layer and retried indefinitely -- never a security error -- because a rolling upgrade can leave one node ahead. A durable-ack capability gap is terminal and fails fast, since retrying cannot turn a non-primary into a primary. Records that a batch fitting no split needs its own error class, not a message match: it is retained for a larger-cap node, close() must recognise and discard it rather than abandon already-published rows, and reset() discards it. Notes QwpServerInfo and QwpBatchBuffer are egress-only, so they are not ported. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 92 ++++++++++++++++++- 1 file changed, 90 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 582b4b4..ad49360 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -25,6 +25,11 @@ Each of these is a separate future spec: - multi-host HA failover (`failover_*` keys, roles and zones); - the UDP sender. +Two Java classes look ingest-relevant and are not — do not port them here: +`QwpServerInfo` / `QwpServerInfoDecoder` decode a `SERVER_INFO` frame sent by a +QWP **egress** server as its first frame after the upgrade; the ingest path never +receives one. `QwpBatchBuffer` is likewise egress (`RESULT_BATCH` decode). + ### 1.2 Single endpoint — and what that degrades Because multi-host failover is out of scope, this stack connects to **one** @@ -274,6 +279,9 @@ same value a successful flush leaves behind, and read as an empty delta) and 1.3.7's "Return never-shipped symbol ids on reset()" does, and omitting it produces a permanently wedged sender rather than a visible error. +`reset()` is also the documented recovery from a batch that fits no split (5.1): +it discards the retained oversized batch and leaves the sender usable. + ### 4.1.1 A throwing column setter must roll back the row This is a columnar-specific invariant with no ILP analogue, and it is easy to @@ -391,6 +399,23 @@ failover *between* the reads sizes frames against different caps and breaks the all-or-nothing guarantee. In Node every `await` inside the flush is exactly that failover window, so this must be a local variable, not a field read. +**A batch that fits no split needs its own error type.** When a batch cannot fit +the cap *however* it is divided, Java raises a distinct +`BatchTooLargeForCapException` rather than a generic error, and the distinction +is load-bearing rather than cosmetic: + +- the batch is **retained**, so it can still go out later against a larger-cap + node; +- `close()` must **recognise the type and discard the batch**, because letting it + escape would skip commit and drain and abandon every row an earlier successful + flush already published (8.3.1); +- `reset()` discards the retained batch and leaves the sender usable — the + non-destructive recovery; +- Java notes explicitly that matching on the message text instead would silently + swallow unrelated failures. + +So this must be a real error class in the Node port, not a string check. + **The split is deliberately not atomic across frames.** A publish failure at frame `k > 1` (backpressure deadline, recycle timeout) leaves frames `1..k-1` on the ring as deferred-but-uncommitted. The error propagates past the @@ -664,6 +689,24 @@ The `421` rule is why §8.4 can say a transient all-replica window is never quarantined on a wall-clock budget: connect-time role rejects retry forever by design. +**A successful 101 can still fail**, in two ways the above table does not cover: + +- **Unsupported `X-QWP-Version`.** The upgrade completed but the server + advertised a version outside our range. This is **transient at every layer** + and must never be classified as a security error: a rolling upgrade can leave + one node ahead of its peers. The background reconnect loop retries + indefinitely; a blocking initial connect consumes its retry budget and only + then surfaces an error. Implementing "unsupported version" as fatal — the + obvious reading — is backwards. +- **Durable-ack capability gap.** `request_durable_ack=on` but the server did not + echo `X-QWP-Durable-Ack: enabled` (or every endpoint role-rejected, so no + primary was reached). This one **is terminal and fails fast**: retrying the + same endpoints will not turn a non-primary into a durable-ack-capable primary, + so it must not burn the reconnect budget. + +Note the asymmetry — version mismatch retries forever, durable-ack mismatch +fails immediately — and that both arrive *after* a successful handshake. + ### 6.5.2 TLS — `tls_roots` does not port directly `tls_verify` maps cleanly: `on` → default verification, `unsafe_off` → @@ -684,6 +727,47 @@ are not supported by the Node client; convert to PKCS#12 or PEM" — not a parse error. A connect string that works against Java may therefore fail here, so this belongs in the README's compatibility notes, not only in this spec. +### 6.5.3 Per-column accumulation rules (PR 4 / PR 5) + +These live in `QwpTableBuffer.ColumnBuffer` and are the substance of PRs 4 and 5. +None are inferable from the wire format alone. + +**Row and column lifecycle** + +- **Columns are created on first sight and their type is locked.** A later value + of a different type for the same name throws a type-mismatch error naming both + types. Column count is capped at `MAX_COLUMNS_PER_TABLE` (2048) at creation. +- **Column names** must be non-empty, pass QuestDB's valid-name check, and be + ≤ 127 bytes. "Too long" and "illegal characters" are *distinct* errors. +- **Duplicate column within one row: first value wins, silently.** If a column + already holds a value for the in-progress row, the second write is ignored with + no error — matching ILP server behaviour. Detected as + `column.size > rowCount`. +- **Row completion back-fills nulls.** At end-of-row every column that did not + receive a value this row has a null appended, so all columns stay at equal + length. This is the mechanism behind the null bitmap (6.2.1), and it is the + same invariant that a throwing setter must not break (4.1.1). +- **Per-row size guard.** After back-filling, a row whose encoded size exceeds + the server batch cap throws "row too large for server batch cap". This is a + *per-row* check, separate from the per-frame split in 5.1 — a single row larger + than the cap can never be sent by any split, so it fails early rather than + wedging the splitter. + +**Type-specific rules** + +| Type | Rule | +|---|---| +| GEOHASH | Precision is 1–60 and **locked on the column's first value**; a differing precision throws. This is why the wire carries one precision varint per column (6.3). | +| DECIMAL64/128/256 | Scale is **locked on the first value**, and a later value with a different scale is **automatically rescaled** to the column's scale — not rejected. Rescaling throws if it would lose precision, or if the result exceeds the type's capacity (e.g. "Decimal128 overflow"). | +| BINARY | A null must be expressed via the null bitmap, never as a null value or negative length — both throw. A non-empty value with a zero pointer throws. | +| DOUBLE_ARRAY / LONG_ARRAY | Shapes must be regular; jagged input throws "irregular array shape". Supplying more values than the declared shape throws. Total element count must fit in an int. | +| VARCHAR / BINARY | Aggregate string data is capped at **2 GiB per batch**; the error tells the caller to flush more frequently. | +| SYMBOL | A column may not mix global symbol ids with local dictionary values; doing so throws. This falls out of the two dictionary modes (5.2) — the mode is per-connection, so a column must not straddle it. | + +The decimal auto-rescale is the one most likely to be mis-ported as a simple +lock-and-reject. It changes user-visible behaviour: writing `1.5` then `1.25` to +a scale-1 column is an error, while `1.25` then `1.5` is not. + ### 6.6 Server responses ``` @@ -1282,8 +1366,8 @@ could actually use the feature. | 1 | `ws/`: framing, masking, handshake, upgrade-failure classification (6.5.1), TLS mapping (6.5.2), net/tls socket | unit + mock server | | 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | | 3 | Sender wiring: `ws://` config (4 sites, 3.5), `QwpBuffer`/`QwpTransport`, byte + interval auto-flush, cap-split (5.1) | **testcontainers e2e green** | -| 4 | Remaining scalar types + null bitmap | golden + e2e | -| 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 | golden + e2e | +| 4 | Remaining scalar types, null bitmap, row lifecycle rules (6.5.3) | golden + e2e | +| 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 + their per-type rules (6.5.3) | golden + e2e | | 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | | 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | | 8 | defer-commit + commit frame (5.1.1) + zstd (feature-detected) | e2e both on and off | @@ -1327,6 +1411,10 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Decimal scale mis-ported as lock-and-reject.** Java *rescales* to the + column's scale and only throws on precision loss or capacity overflow (6.5.3). + A lock-and-reject port rejects data Java accepts, and the asymmetry is + order-dependent — `1.25` after `1.5` differs from `1.5` after `1.25`. - **A throwing setter desynchronising the columns.** Unequal per-column lengths (4.1.1) corrupt every subsequent frame from that table buffer while each frame still looks structurally valid. Tests must throw from a setter mid-row — first From b04a237a0cca8d7412c80240f22949b23a221627 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:37:17 +0100 Subject: [PATCH 015/121] docs: describe the staging buffer swap and close the class coverage matrix Fifteenth pass, over the last unreferenced ingest classes. Adds 5.3, the staging buffer. Sections 5.1 and 8.3.1 both said "seal and swap the buffer" without ever saying what is swapped. Java stages encoded messages in a MicrobatchBuffer between the encoder and the ring, keeps two of them, and cycles FILLING -> SEALED -> SENDING -> RECYCLED. Records the six-step swap, including the early return on an empty buffer (which is part of why the commit frame cannot trust batch state) and the 30s recycle wait that 5.1 already referenced as a mid-split failure cause without defining. Makes the Node decision explicit rather than leaving it implied: the second buffer exists because the first stays pinned while read asynchronously after handoff, and in Node that hazard recurs at every await inside append. Either copy on append and drop the swap entirely, or port the two-buffer wait. Recommends copy-on-append, since Node must copy into a Buffer for the write regardless, and notes that choosing it makes the 30s timeout unreachable and that 5.1 should then stop naming it. Completes the coverage matrix in 1.1. RowView, ColumnView, RowCallback, QwpColumnBatch, QwpColumnLayout, QwpBindValues, QwpBindSetter, QueryEvent, QwpEgressIoThread and QwpResultBatchDecoder are result-batch decode and belong to the query spec. QwpSpscQueue, NativeBufferWriter, SegmentedNativeBufferWriter, NativeSegmentList and OffHeapAppendMemory are threading and off-heap allocation with no Node analogue and no protocol semantics. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 55 +++++++++++++++++-- 1 file changed, 51 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index ad49360..685cd4e 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -25,10 +25,22 @@ Each of these is a separate future spec: - multi-host HA failover (`failover_*` keys, roles and zones); - the UDP sender. -Two Java classes look ingest-relevant and are not — do not port them here: -`QwpServerInfo` / `QwpServerInfoDecoder` decode a `SERVER_INFO` frame sent by a -QWP **egress** server as its first frame after the upgrade; the ingest path never -receives one. `QwpBatchBuffer` is likewise egress (`RESULT_BATCH` decode). +Several Java classes look ingest-relevant and are not — do not port them here: + +- `QwpServerInfo` / `QwpServerInfoDecoder` decode a `SERVER_INFO` frame sent by a + QWP **egress** server as its first frame after the upgrade; the ingest path + never receives one. +- `QwpBatchBuffer`, `QwpColumnBatch`, `RowView`, `ColumnView`, `RowCallback`, + `QwpColumnLayout`, `QwpBindValues`, `QwpBindSetter`, `QueryEvent`, + `QwpEgressIoThread`, `QwpResultBatchDecoder` are all result-batch decode and + belong to the query spec. +- `QwpSpscQueue`, `NativeBufferWriter`, `SegmentedNativeBufferWriter`, + `NativeSegmentList`, `OffHeapAppendMemory` are threading and off-heap + allocation machinery with no Node analogue: the SPSC queue exists to hand + frames between the producer and I/O threads, which the event loop makes + unnecessary, and the rest are replaced wholesale by `Buffer`. Neither carries + protocol semantics. +- `QwpHostHealthTracker` is multi-host (1.1, 1.2). ### 1.2 Single endpoint — and what that degrades @@ -477,6 +489,41 @@ The mode is therefore a consequence of what durable state exists, not a user toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict mode — which is what makes PR 6 shippable before PR 12. +### 5.3 The staging buffer and its swap + +Sections 5.1 and 8.3.1 refer to "seal and swap the buffer" without saying what +is swapped. Java stages encoded messages in a `MicrobatchBuffer` between the +encoder and the SF ring, with a four-state lifecycle — +`FILLING → SEALED → SENDING → RECYCLED` — and keeps **two** of them. + +`sealAndSwapBuffer` does, in order: + +1. return immediately if the active buffer holds no data (this is why an empty + flush publishes nothing, and part of why the commit frame cannot rely on + batch state being reset — 5.1.1); +2. seal the active buffer; +3. swap the *other* buffer in as active; +4. **if that buffer is still in use, wait for it to be recycled**, bounded at + 30 s, throwing "Timeout waiting for buffer to be recycled" on expiry — this is + the buffer-recycle timeout 5.1 names as a mid-split failure cause; +5. reset the newly active buffer; +6. mark the sealed buffer `SENDING` and hand it to the engine, which appends and + returns once published. + +The buffer stays pinned until the wire send completes, which is what the second +buffer is for: the producer keeps filling while the previous batch is in flight. + +**Node decision.** The pin exists because the buffer is read asynchronously after +handoff. In Node the same hazard appears at every `await` inside append, so +choose one and state it: either **copy on append** — the segment write takes +ownership of its own bytes, one staging buffer suffices, and the swap machinery +disappears — or **port the two-buffer swap** with an equivalent bounded wait. +Copy-on-append is recommended: Node must copy into a `Buffer` for the write +anyway, so the copy Java avoids is not actually avoidable here, and it removes +a timeout error path rather than reproducing it. If copy-on-append is chosen, +the 30 s recycle timeout referenced in 5.1 becomes unreachable and should be +dropped from that list rather than left as dead prose. + ## 6. Wire format All little-endian, byte-level. From 94193ad154525382f96a72f0263c4e66151363dc Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:38:56 +0100 Subject: [PATCH 016/121] docs: correct notification inbox policy and the close() suppression test Sixteenth pass, over SenderErrorDispatcher and SenderConnectionDispatcher, which section 4.2 had captured only at javadoc level. The inbox drops the OLDEST entry, not the newest. Section 4.2 said only "surplus dropped", and the intuitive bounded-queue implementation -- reject the newcomer when full -- is backwards. Watermarks are monotonic, so the newest entry is always the most informative and dropping the head compresses information rather than losing it; the spec mandates it, and it needs a deque rather than a plain queue. Under load a drop-newest port would retain stale notifications and discard current state. Added to the risks with a test that fills the inbox and asserts the newest survives. Corrects the capacity: default is 256, minimum 16. The spec had quoted the minimum as though it were the whole story and never gave a default. Records that the dispatcher starts lazily on first delivery, that handlers are swappable after connect by design, and that close drains under a short deadline rather than discarding. Narrows the close() suppression test in 8.3.1. Java tracks both "a custom handler ever received any error" and "it received THE latched terminal error", and close() consults only the second. Gating on the first would let a routine RETRIABLE rejection delivered earlier suppress the close-time report of a later, genuinely unsurfaced TERMINAL error. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 33 ++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 685cd4e..3b1834e 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -335,11 +335,23 @@ They share one delivery contract that must survive the port: dispatcher thread so a slow handler cannot stall publishing or reconnect. Node has no such thread, so callbacks must be dispatched via a queue drained on a `setImmediate`-style tick — never called inline from the socket handler. -- **Bounded inbox, surplus dropped.** Capacity comes from `error_inbox_capacity` - and `connection_listener_inbox_capacity` (minimum **16**), and drops are - counted and readable. Without the bound, a slow user callback becomes unbounded - memory growth. +- **Bounded inbox that drops the OLDEST.** Capacity is `error_inbox_capacity` / + `connection_listener_inbox_capacity`, **default 256**, minimum **16**. When + full, the producer drops the **head** to admit the new entry — it never + blocks, spins, or rejects the newcomer. Drop-newest is the intuitive + implementation and is **wrong**: watermarks are monotonic, so the newest entry + is always the most informative, and dropping the oldest *compresses* + information instead of losing it. This needs a deque, not a plain queue. + Drops are counted and readable so a non-zero count tells an operator the + handler is too slow. +- **Started lazily**, on first delivery, so a workload that never errors pays + nothing. In Node that means not creating the dispatch machinery until the + first notification. +- **Handlers are swappable after connect**, deliberately — installing one is not + a pre-connect-only concern. - **Handler exceptions are caught and logged**; the sender keeps running. +- **Close drains** remaining entries under a short deadline (100 ms in Java) + rather than discarding them. - **Success connection events fire on every transition; failure events may be coalesced** under inbox pressure. `AUTH_FAILED` fires *before* the corresponding error is observable on the producer side. @@ -1244,6 +1256,14 @@ an error instance the user already caught from an earlier call. Java snapshots t already-surfaced error once, precisely so a terminal latched between two reads cannot be misattributed as user-owned and silently dropped. +The suppression test is narrower than it looks, and getting it wrong disables the +safety net. Java tracks **two** facts: whether a custom handler ever received +*any* error, and whether it received **the** terminal error — the exact one the +I/O loop latched. `close()` consults only the second. Using the first would let a +routine `RETRIABLE` rejection delivered minutes earlier suppress the close-time +report of a later, genuinely unsurfaced `TERMINAL` error. "Any error ever" is too +coarse a signal to gate this on. + Deferred commits interact here: frames above the last commit-bearing (non-`DEFER_COMMIT`) FSN belong to a transaction whose commit was never published, so the server will never ACK them. Close-time drain must target the @@ -1352,6 +1372,7 @@ not have yet: | `reconnect_max_duration_millis` | 300,000 | | `catch_up_cap_gap_min_escalation_window_millis` | 300,000 | | segment manager poll tick | 1 ms | +| `error_inbox_capacity` / `connection_listener_inbox_capacity` | 256 (minimum 16) | | catch-up packing limit when cap unadvertised | 64 KiB | | max catch-up cap-gap attempts (orphan drainer only) | 16 | @@ -1458,6 +1479,10 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Notification inbox dropping the wrong end.** Drop-newest is the intuitive + bounded-queue policy and inverts the intent (4.2): under load the handler would + keep stale entries and discard the current state. Tests must fill the inbox and + assert the *newest* notification survives. - **Decimal scale mis-ported as lock-and-reject.** Java *rescales* to the column's scale and only throws on precision loss or capacity overflow (6.5.3). A lock-and-reject port rejects data Java accepts, and the asymmetry is From 52f643e1ff8fa01dbb32dbd2ccfaf500aac2f26b Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:42:35 +0100 Subject: [PATCH 017/121] docs: correct per-dispatcher capacities and complete the event kind list Seventeenth pass, over SenderConnectionDispatcher and SenderProgressDispatcher. Fixes an error introduced by the previous pass. Collapsing error_inbox_capacity and connection_listener_inbox_capacity into one table row asserted a shared default of 256. The three dispatchers have different defaults: errors 256, progress 256, connection events 64 -- deliberately smaller because connection events are sparse next to per-batch server errors. The connect-string minimum of 16 applies to the configurable pair; the dispatcher constructor itself only requires >= 1. Completes the connection event list. Kind has seven members and section 4.2 omitted DISCONNECTED. Records the event payload, which the spec had reduced to a bare kind: host and port, the previous host and port, an attempt number, a round number, a cause and a timestamp. The attempt/round pair is what makes a reconnect storm diagnosable. AUTH_FAILED carries the upgrade auth failure as its cause, which is the path by which the terminal credential case reaches the listener before the producer-side throw. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 3b1834e..d835319 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -326,7 +326,7 @@ Java exposes three separate surfaces; the Node port mirrors all three: | Java | Fires on | |---|---| | `SenderErrorHandler` | rejections — carries category, policy, `fromFsn`/`toFsn`, `quarantinedPath` | -| `SenderConnectionListener` | `CONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` — two of these are never emitted single-endpoint (1.2) | +| `SenderConnectionListener` | seven kinds: `CONNECTED`, `DISCONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` — two are never emitted single-endpoint (1.2) | | `SenderProgressHandler` | the ACK watermark advancing | They share one delivery contract that must survive the port: @@ -335,8 +335,10 @@ They share one delivery contract that must survive the port: dispatcher thread so a slow handler cannot stall publishing or reconnect. Node has no such thread, so callbacks must be dispatched via a queue drained on a `setImmediate`-style tick — never called inline from the socket handler. -- **Bounded inbox that drops the OLDEST.** Capacity is `error_inbox_capacity` / - `connection_listener_inbox_capacity`, **default 256**, minimum **16**. When +- **Bounded inbox that drops the OLDEST.** Capacities differ per dispatcher and + are not interchangeable: errors **256**, progress **256**, connection events + **64** — connection events are sparse compared with per-batch server errors. + The connect-string minimum is 16. When full, the producer drops the **head** to admit the new entry — it never blocks, spins, or rejects the newcomer. Drop-newest is the intuitive implementation and is **wrong**: watermarks are monotonic, so the newest entry @@ -356,6 +358,13 @@ They share one delivery contract that must survive the port: coalesced** under inbox pressure. `AUTH_FAILED` fires *before* the corresponding error is observable on the producer side. +A connection event is not just a kind. It carries the host and port, the +**previous** host and port, an attempt number, a round number, a cause, and a +timestamp — the attempt/round pair is what makes reconnect storms diagnosable, so +carry it even though single-endpoint (1.2) pins the round. `AUTH_FAILED`'s cause +is the auth failure from the upgrade (6.5.1), which is how the terminal +credential case reaches the listener before the producer-side throw. + Progress-handler semantics are narrower than they look: the watermark advances **only on server OK frames** — a rejection never advances it — values are strictly increasing, and one call may skip several FSNs when the server batches @@ -1372,7 +1381,9 @@ not have yet: | `reconnect_max_duration_millis` | 300,000 | | `catch_up_cap_gap_min_escalation_window_millis` | 300,000 | | segment manager poll tick | 1 ms | -| `error_inbox_capacity` / `connection_listener_inbox_capacity` | 256 (minimum 16) | +| `error_inbox_capacity` | 256 (config minimum 16) | +| `connection_listener_inbox_capacity` | **64** (config minimum 16) | +| progress inbox capacity | 256 | | catch-up packing limit when cap unadvertised | 64 KiB | | max catch-up cap-gap attempts (orphan drainer only) | 16 | From 63bf938a5ff4cd48ff28152919e99eaa064ba1ff Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 19:13:40 +0100 Subject: [PATCH 018/121] docs: add config value grammars and correct the auto_flush_bytes default Eighteenth pass, over the per-key value parsing that ConfigSchema defers to Sender's own helpers. The spec mirrored key names without their grammars. Corrects auto_flush_bytes. The spec claimed a default of 8 MiB, taken from QwpWebSocketSender.DEFAULT_AUTO_FLUSH_BYTES, but the builder's WebSocket default is 0 -- which is exactly what auto_flush_bytes=off sets. Byte-based auto-flush is therefore OFF by default and the trigger is rows and interval only; defaulting to 8 MiB would flush on a trigger Java does not use. Records that an explicit off is preserved even once the server advertises a cap, and that the per-row guard is what makes opting out safe. Adds 9.1.1, the value grammars. Byte counts accept k/m/g and also t (which the javadoc omits but the code handles), with an optional trailing b, case-insensitive, 1024-based. A port reaching for parseInt reads auto_flush_bytes=64m as 64 bytes -- a flush per row, with no error raised. Added to the risks as the most likely silent misconfiguration in the config surface. Enum values are case-insensitive. Adds the SF size defaults the spec lacked: sf_max_segment_bytes 4 MiB, sf_max_total_bytes mode-dependent at 128 MiB memory and 10 GiB disk, sf_sync_interval_millis 5000, max_background_drainers 4, max_name_len 127. Records slot naming in 8.3: a slot is // with sender_id defaulting to "default", so a second sender sharing sf_dir without setting it fails with "sf slot already in use". The error must name sender_id as the fix. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 55 ++++++++++++++++++- 1 file changed, 52 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index d835319..c2ba3ae 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -1213,6 +1213,13 @@ the cost model rather than just the mechanism): ### 8.3 Slot locking — two locks, not one +A slot is `//`, and `sender_id` defaults to `"default"`. That +default is fine for a single sender and is a **footgun for two**: a second sender +sharing `sf_dir` without its own `sender_id` fails at startup with "sf slot +already in use". That is the intended behaviour (8.1's single-producer rule), so +the error must name `sender_id` as the fix rather than reading as a mystery +lock conflict. + Java's `SlotLock` provides **two distinct advisory locks**, and the second is not optional — the orphan-adoption sequence depends on it: @@ -1369,8 +1376,14 @@ not have yet: | Setting | QWP default | |---|---| | `auto_flush_rows` | 1,000 | -| `auto_flush_bytes` | 8 MiB | +| `auto_flush_bytes` | **off (0)** — see below | | `auto_flush_interval` | 100 ms | +| `sf_max_segment_bytes` | 4 MiB | +| `sf_max_total_bytes` | **mode-dependent**: 128 MiB memory, 10 GiB disk | +| `sf_sync_interval_millis` | 5,000 | +| `max_background_drainers` | 4 | +| `max_name_len` | 127 | +| `sender_id` | `"default"` | | `auth_timeout_ms` | 15,000 | | background connect timeout | 15,000 ms | | `max_frame_rejections` | 4 | @@ -1387,8 +1400,41 @@ not have yet: | catch-up packing limit when cap unadvertised | 64 KiB | | max catch-up cap-gap attempts (orphan drainer only) | 16 | -`auto_flush_bytes` must additionally be clamped to the server-advertised -`X-QWP-Max-Batch-Size` (default 16 MiB) once the handshake completes. +**Byte-based auto-flush is off by default on WebSocket.** The builder's WS +default is `0`, which is exactly what `auto_flush_bytes=off` sets — so the +trigger is rows and interval only. `QwpWebSocketSender.DEFAULT_AUTO_FLUSH_BYTES` +(8 MiB) is *not* the effective default; defaulting to it would make the Node +client flush on a trigger Java does not use. + +When a byte trigger *is* set, it is clamped to the server-advertised +`X-QWP-Max-Batch-Size` (default 16 MiB) after the handshake. But an explicit +`off` is **preserved** even when the server advertises a cap — an application +that opted out keeps the contract it asked for. Oversize rows are still caught +by the per-row guard against `serverMaxBatchSize` (6.5.3), which is what makes +opting out safe. + +### 9.1.1 Value grammars — `ConfigSchema` does not define these + +`ConfigSchema` registers most ingest keys as plain strings and leaves the value +grammar to the sender's own parsers. Mirroring the key *names* without the +grammars produces silent misconfiguration. + +**Byte-count values** (`auto_flush_bytes`, `sf_max_total_bytes`, +`sf_max_segment_bytes`, buffer sizes) accept a plain decimal or a unit suffix: + +- `64k` / `64kb`, `64m` / `64mb`, `4g` / `4gb`, and `2t` / `2tb` + — note the code handles `t`, though its own javadoc lists only k/m/g; +- suffixes are **case-insensitive**, and a trailing `b`/`B` is stripped first; +- multipliers are **powers of 2** (1024-based), not 1000; +- blank is rejected with " cannot be empty"; a bare suffix is rejected. + +A Node port that reaches for `parseInt` reads `auto_flush_bytes=64m` as **64 +bytes** — a flush per row, with no error. That is the single most likely silent +misconfiguration in the whole config surface. + +**Enum values** are case-insensitive: `sf_durability` ∈ {`memory`, `periodic`, +`flush`, `append`}, rejected with the allowed set named. `auto_flush` and the +other on/off keys take `on`/`off`. ### 9.2 zstd and the Node version floor @@ -1490,6 +1536,9 @@ could actually use the feature. alone turns brief outages into producer-fatal terminals. The mock-server tests must include a "4 strikes inside the dwell window" case that asserts *no* escalation. +- **Size suffixes parsed as plain integers.** `auto_flush_bytes=64m` read by + `parseInt` is 64 bytes, not 64 MiB (9.1.1) — a flush per row, silently. Config + tests must cover every suffix form, including the undocumented `t`. - **Notification inbox dropping the wrong end.** Drop-newest is the intuitive bounded-queue policy and inverts the intent (4.2): under load the handler would keep stale entries and discard the current state. Tests must fill the inbox and From ead61b2d4492574cb1db7b5b84d0263ef53b08b8 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 19:16:11 +0100 Subject: [PATCH 019/121] docs: add cross-key validation and correct the sf_durability mode count Nineteenth pass, over LineSenderBuilder.build()'s validation, which the spec had none of. Corrects sf_durability. Section 8.2 said it selects one of exactly four modes. Four values parse, but build() rejects flush and append as "not yet supported (use sf_durability=memory or periodic)", so only two are usable. The parser should still accept all four so the error names the right cause. Adds 9.2. Every Side.INGRESS key is WebSocket-only and throws when combined with http:: or tcp::, not only the two keys 8.2 mentioned. Three keys are rejected the other way, for WebSocket: protocol_version, a disabled auto-flush interval, and ILP max_backoff. The protocol_version rule is stronger than 3.5 assumed -- Java makes an explicitly supplied value an error under ws::, which turns the silent ILP-fallback hazard into a loud failure, so 3.5 now points at it. Records the dependency rules: periodic durability requires sf_dir, sf_sync_interval_millis requires periodic, drain_orphans requires sf_dir, tls_roots cannot combine with tls_verify=unsafe_off, and WebSocket requires at least one host:port pair. Records that mode selection is implicit: there is no store_and_forward key, sf_dir present means disk mode and absent means memory mode. That single fact drives memory-vs-disk throughout section 8, including the mode-dependent sf_max_total_bytes default, and the spec never stated it. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 61 ++++++++++++++++--- 1 file changed, 53 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index c2ba3ae..4a1bf57 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -226,6 +226,8 @@ Both error strings are very likely asserted verbatim in `sender.config.test.ts`, so PR 3 touches those tests. **Hazard — `createBuffer` must branch on protocol before `protocol_version`.** +(Java's stronger rule — reject an explicitly supplied `protocol_version` under +`ws::` — is in 9.2 and should be adopted alongside this.) `parseProtocolVersion` has a `default:` arm assigning `PROTOCOL_VERSION_V1` to any protocol that is not HTTP/HTTPS. A `ws::` sender therefore reaches `createBuffer` carrying `protocol_version = 1`, and `createBuffer` switches on @@ -1193,12 +1195,12 @@ unlinking, and fsyncs it **again** after the batch. Close uses the same covering order, so the durable watermark always guards any acknowledged segment a host crash restores. -`sf_durability` selects one of exactly four modes — `memory`, `periodic`, -`flush`, `append` — and any other value is rejected. `sf_sync_interval_millis` -sets the periodic barrier. Both keys are **WebSocket-only**: Java throws -`"sf_durability is only supported for WebSocket transport"` if they appear with -another protocol, and the Node port must reject them the same way for `http::` -and `tcp::`. +`sf_durability` **parses** four values — `memory`, `periodic`, `flush`, +`append` — but only **two are usable**: `build()` rejects `flush` and `append` +with "not yet supported (use sf_durability=memory or periodic)". Implement all +four in the parser so the error is the right one, and reject the reserved pair at +construction. `sf_sync_interval_millis` sets the periodic barrier. See 9.2 for +the cross-key rules that bind these together. **Two consequences of the Node primitives** (expected deviations, but they change the cost model rather than just the mechanism): @@ -1356,7 +1358,7 @@ ignore". **There is no `zstd` configuration key.** `zstd` is an enum *value* of the egress-side `compression` key (`zstd` | `raw` | `auto`). Ingest-side zstd is -therefore purely a handshake negotiation (9.2) with no connect-string control — +therefore purely a handshake negotiation (9.3) with no connect-string control — do not invent a key for it. ### 9.1 Defaults differ from ILP — do not inherit the ILP ones @@ -1436,7 +1438,50 @@ misconfiguration in the whole config surface. `flush`, `append`}, rejected with the allowed set named. `auto_flush` and the other on/off keys take `on`/`off`. -### 9.2 zstd and the Node version floor +### 9.2 Cross-key validation + +Java validates combinations at construction, not just individual values. The spec +previously had none of this, and several rules are load-bearing. + +**Every ingest QWP key is WebSocket-only.** Not just `sf_durability` and +`sf_sync_interval_millis` as 8.2 implied — the whole `Side.INGRESS` set throws +" is only supported for WebSocket transport" when combined with `http::` or +`tcp::`. Reproduce this per key so a misplaced key names itself. + +**Keys rejected *for* WebSocket** — these are ILP-only and must error, not be +silently ignored: + +| Rejected with `ws::` | Message | +|---|---| +| `protocol_version` | "protocol version is not supported for WebSocket protocol" | +| `auto_flush=off` (interval disabled) | "disabling auto-flush is not supported for WebSocket protocol" | +| ILP `max_backoff` | "max backoff is not supported for WebSocket protocol" | + +The `protocol_version` rule is stronger than 3.5 assumed. 3.5 says +`parseProtocolVersion` must leave `ws`/`wss` unset; Java additionally makes an +*explicitly supplied* `protocol_version` an error under `ws::`. Adopt that — it +converts the silent-ILP-fallback hazard into a loud failure. + +Note also that auto-flush cannot be fully disabled on WebSocket, which follows +from 9.1: with `auto_flush_bytes` off by default, the interval and row triggers +are the only ones left. + +**Dependency rules:** + +- `sf_durability=periodic` **requires** `sf_dir`. +- `sf_sync_interval_millis` **requires** `sf_durability=periodic`. +- `drain_orphans` **requires** `sf_dir`. +- `tls_roots` **cannot** be combined with `tls_verify=unsafe_off`. Java's message + names both escapes — "remove tls_verify to use custom roots, or remove + tls_roots to disable certificate validation" — and is worth copying verbatim. +- WebSocket requires at least one `host:port` pair in `addr` (1.2). + +**Mode selection is implicit.** There is no `store_and_forward=on` key: **`sf_dir` +present means disk mode, absent means memory mode.** That single fact drives +memory-vs-disk throughout section 8, including the mode-dependent +`sf_max_total_bytes` default (9.1), and the spec never stated it. + +### 9.3 zstd and the Node version floor `node:zlib`'s `zstdCompress` landed in **Node 22.15.0**. The client's documented floor is Node 20 and CI runs `[20, 22, latest]`. A naive "require Node 22" rule From 26920e5b23a0de4e9aa98af56e0a5cd8d400c9e6 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 19:20:29 +0100 Subject: [PATCH 020/121] docs: add connect timing and precise sf_durability semantics Twentieth pass, over InitialConnectMode and SfDurability. Adds 4.3, connect timing, which the spec never addressed -- it described flush and reconnect without ever saying when the initial connection happens. Java has OFF, SYNC and ASYNC, and the default is DERIVED: setting any reconnect_* key implicitly upgrades construction from non-connecting to connecting-with-retry. The reason is that the reconnect_* knobs read as a generic retry budget while the underlying path governs only reconnects from an established connection, so a user who sets a budget and gets no retry on the first connect has hit what Java calls the canonical footgun. Porting the three modes with a fixed default would reintroduce it. Notes that initial_connect_mode is builder-only while initial_connect_retry is the connect-string key. Replaces 8.2's vague "sf_durability governs when fdatasync runs" with the actual semantics: memory never fsyncs explicitly and survives a process crash but not an OS or power crash; periodic checkpoints in the background at a target cadence that is not a bound, since scheduler and storage latency add to the real power-loss window; flush and append are reserved. Records the consequence the spec had obscured: disk mode alone is not power-loss durability. sf_dir selects disk mode but durability still defaults to memory, so files are written and never explicitly synced. Power-loss survival needs sf_durability=periodic, which itself requires sf_dir. This is the same property 8.1.5 records for .symbol-dict, and under the default it applies to the segments too. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 53 ++++++++++++++++--- 1 file changed, 46 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 4a1bf57..5c3340a 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -378,7 +378,31 @@ server-side *commit*, not object-store durability. Anything gating downstream side effects on durability must opt into `request_durable_ack`. This distinction belongs in the README, not just here. -### 4.3 Flush semantics +### 4.3 Connect timing — and a default that changes it silently + +The spec has said nothing about *when* the initial connection happens. Java has +three modes — `OFF`, `SYNC`, `ASYNC` — and, crucially, **the default is derived +from other keys**: + +- if any `reconnect_*` key is set (`reconnect_max_duration_millis`, + `reconnect_initial_backoff_millis`, `reconnect_max_backoff_millis`) → **SYNC**: + construction connects, retrying under that budget; +- otherwise → **OFF**: construction does not connect. + +The reasoning is worth keeping. The `reconnect_*` knobs read as a generic retry +budget, but the underlying path governs only reconnects *from an already +established connection*. A user who sets a retry budget and then gets no retry on +the very first connect has hit what Java calls "the canonical footgun"; the +implicit upgrade to `SYNC` removes it. + +So setting a `reconnect_*` key changes construction from non-connecting to +connecting. Port the derivation, not just the modes — implementing the three +modes with a fixed default reintroduces exactly the footgun the derivation +exists to remove. `initial_connect_mode` itself is builder-only and is **not** a +connect-string key; `initial_connect_retry` is the connect-string key in this +area. + +### 4.4 Flush semantics `flush()` resolves once the frame is **published into the store-and-forward engine** — in RAM for memory mode, on disk for disk mode. It does *not* wait for @@ -1195,12 +1219,27 @@ unlinking, and fsyncs it **again** after the batch. Close uses the same covering order, so the durable watermark always guards any acknowledged segment a host crash restores. -`sf_durability` **parses** four values — `memory`, `periodic`, `flush`, -`append` — but only **two are usable**: `build()` rejects `flush` and `append` -with "not yet supported (use sf_durability=memory or periodic)". Implement all -four in the parser so the error is the right one, and reject the reserved pair at -construction. `sf_sync_interval_millis` sets the periodic barrier. See 9.2 for -the cross-key rules that bind these together. +`sf_durability` **parses** four values but only **two are usable**: + +| Value | Meaning | +|---|---| +| `memory` | **Default.** Never fsync explicitly — bytes live in the OS page cache. Survives a **process** crash, **not** an OS or power crash. Lowest latency. | +| `periodic` | Background checkpoint of published frames every `sf_sync_interval_millis`. The interval is a *target cadence*: scheduler and storage latency add to the real power-loss window, so it is not a bound. | +| `flush` | Reserved for a future synchronous `flush()` barrier — rejected by `build()`. | +| `append` | Reserved for a future per-append barrier — rejected by `build()`. | + +Implement all four in the parser so a user of the reserved pair gets "not yet +supported (use sf_durability=memory or periodic)" rather than "unknown value", +and reject them at construction. + +**Disk mode alone is not power-loss durability.** `sf_dir` selects disk mode +(9.2), but durability still defaults to `memory` — files are written and never +explicitly synced. Power-loss survival requires `sf_durability=periodic`, which +in turn requires `sf_dir`. Section 8.1.5's note that `.symbol-dict` is +"page-cache durable, not host-crash durable" is the same property, and it applies +to the segments too under the default. + +See 9.2 for the cross-key rules binding these together. **Two consequences of the Node primitives** (expected deviations, but they change the cost model rather than just the mechanism): From 267aec86f6fc22f9afdf9c4b87cc0afa6ad4abb5 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 19:22:25 +0100 Subject: [PATCH 021/121] docs: add quarantine mechanics, the unset sentinel rule, and missing defaults Twenty-first pass, over the remaining builder constants. Corrects 8.4's account of quarantine. Setting a torn slot aside is two steps, not one: the slot is RENAMED with a quarantine infix -- deliberately not the sender's own slot name, so a restarting sender cannot re-adopt it as its working slot -- and then marked .failed so the orphan drainer skips it too. The spec had only the sentinel, which stops the drainer but not the owner. Adds the cap: at most 64 quarantined copies of one slot before construction refuses another, since each is an unreplayable slot a human must inspect and unbounded accumulation turns a disk-space problem into a second incident. Adds 9.1.2. Java threads a not-set-explicitly sentinel through every numeric option rather than pre-seeding defaults, so that an explicitly supplied value equal to the default still fails fast. This is a constraint on the port, not a Java idiom: the natural JS shape collapses "unset" and "set to the default", and both the ws:: rejection rules in 9.2 and the connect-mode derivation in 4.3 key on whether a value was supplied rather than on what it is. Records that close bounds the wait and not the connect -- a close racing an in-flight connect cancels it rather than waiting it out -- and adds the defaults the spec lacked: durable_ack_keepalive_interval_millis 200, close shutdown await 30000, quarantine cap 64. Notes the inbox minimum of 16 is sized to exceed the ten error categories so drop-oldest cannot erase the trailing category distribution. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 47 ++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 5c3340a..3e9de3a 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -340,7 +340,9 @@ They share one delivery contract that must survive the port: - **Bounded inbox that drops the OLDEST.** Capacities differ per dispatcher and are not interchangeable: errors **256**, progress **256**, connection events **64** — connection events are sparse compared with per-batch server errors. - The connect-string minimum is 16. When + The connect-string minimum is 16, chosen so a bursty error stream cannot let + drop-oldest erase the trailing distribution of *categories* — 16 comfortably + exceeds the ten categories in 7.1. When full, the producer drops the **head** to admit the new entry — it never blocks, spins, or rejects the newcomer. Drop-newest is the intuitive implementation and is **wrong**: watermarks are monotonic, so the newest entry @@ -1321,6 +1323,12 @@ routine `RETRIABLE` rejection delivered minutes earlier suppress the close-time report of a later, genuinely unsurfaced `TERMINAL` error. "Any error ever" is too coarse a signal to gate this on. +**Close bounds the wait, not the connect.** The close-time shutdown await +(30 s by default) caps how long `close()` waits for the I/O side to finish its +current send/receive and unwind — it is *not* a connect timeout. A `close()` +racing an in-flight connect **cancels** that connect rather than waiting it out, +so a legitimately slow successful connect is never truncated by this bound. + Deferred commits interact here: frames above the last commit-bearing (non-`DEFER_COMMIT`) FSN belong to a transaction whose commit was never published, so the server will never ACK them. Close-time drain must target the @@ -1341,6 +1349,23 @@ automatic retry, then human-in-the-loop. Abandonment fires `DATA_LOSS` / `ABANDONED` with `quarantinedPath` set. Note that a transient all-replica failover window is **not** terminal and is retried indefinitely. +**Quarantine is a rename plus a sentinel, and it is capped.** Setting a torn slot +aside is two steps, not one: + +1. **Rename** the slot to carry a quarantine infix — deliberately *not* the + sender's own slot name, so a restarting sender does not re-adopt it as its + own working slot; +2. **mark it `.failed`**, so the orphan drainer skips it too. + +Both are needed: the rename stops the owner reclaiming it, the sentinel stops the +drainer replaying it, and between them the bytes stay put for a human to inspect +and resend. + +At most **64** quarantined copies of one slot may accumulate under `sf_dir` +before construction refuses to set aside another. Each is an unreplayable slot +someone must look at, and letting them pile up without bound turns a disk-space +problem into a second incident. + Frames above the last commit-bearing (non-`DEFER_COMMIT`) FSN in a recovered ring belong to a transaction whose commit frame was never published; the server will never ACK them until a later commit covers them. Close-time drain must not @@ -1425,6 +1450,9 @@ not have yet: | `max_background_drainers` | 4 | | `max_name_len` | 127 | | `sender_id` | `"default"` | +| `durable_ack_keepalive_interval_millis` | 200 (≤ 0 disables) | +| close shutdown await | 30,000 | +| max quarantined copies per slot | 64 | | `auth_timeout_ms` | 15,000 | | background connect timeout | 15,000 ms | | `max_frame_rejections` | 4 | @@ -1477,6 +1505,23 @@ misconfiguration in the whole config surface. `flush`, `append`}, rejected with the allowed set named. `auto_flush` and the other on/off keys take `on`/`off`. +### 9.1.2 "Unset" and "set to the default" must stay distinguishable + +Java threads a `PARAMETER_NOT_SET_EXPLICITLY` sentinel through every numeric +option rather than pre-seeding defaults, and the comment states why: it wants to +**fail fast even when an explicitly configured value happens to equal the +default**, because the combination is still a user error and silently accepting +it produces hard-to-debug behaviour. + +This is a real constraint on the Node port, not a Java idiom. The natural JS +shape — `const x = options.x ?? DEFAULT` — collapses "unset" and "set to the +default value" into one state, and several rules in 9.2 depend on telling them +apart: rejecting `protocol_version` under `ws::` must fire even if the user +supplied the value the ILP path would have chosen, and the connect-mode +derivation in 4.3 keys on whether a `reconnect_*` key was *supplied*, not on its +value. Keep options as `undefined`-until-set and resolve defaults at the point of +use. + ### 9.2 Cross-key validation Java validates combinations at construction, not just individual values. The spec From 07b303ba8d22cf69dcc63244566b261bfae957a1 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 19:27:02 +0100 Subject: [PATCH 022/121] docs: add auth precedence rules and the wss-only TLS constraint Twenty-second pass, over build()'s shared prologue. Section 6.5.2 described how tls_verify, tls_roots and tls_roots_password map onto Node's tls.connect but never said they are wss-only. Supplying any of them with plain ws:: throws "tls_verify/tls_roots/tls_roots_password require the wss:: schema" rather than being ignored, and tls_roots_password additionally requires tls_roots. Section 6.5 described the Authorization header as derived from user/password/token without saying how the three interact. They are validated, not inferred: username and password must be supplied together, token is mutually exclusive with both, and the setters are one-shot so configuring either mechanism twice throws "already configured" rather than last-write-wins. Records a cross-client constraint worth honouring: Java deliberately emits the same message text as the egress query client for the username/password rule, so a connect string shared between a sender and a query client fails identically on both sides. Matching those strings keeps a user debugging a shared ws:: string from getting two different diagnoses. Adds the new dependency rules to 9.2 and corrects the key names there and in 6.5 from user to username, the canonical spelling. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 29 +++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 3e9de3a..1e58111 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -758,7 +758,22 @@ accept. Request: `GET /write/v4` with `Sec-WebSocket-Key`, `Sec-WebSocket-Version: 13`, `X-QWP-Client-Id`, `X-QWP-Max-Version: 1`, and `Authorization: Basic|Bearer` -derived from `user`/`password`/`token`. +derived from `username`/`password`/`token`. + +Auth selection is validated, not inferred: + +- `username` and `password` must be supplied **together** — either half alone + throws "username and password must be provided together"; +- `token` is **mutually exclusive** with `username`/`password` — "cannot use both + token and username/password authentication"; +- the setters are one-shot: configuring either mechanism twice throws "already + configured" rather than last-write-wins. + +Java notes it deliberately emits *the same message text* as the egress query +client for the first rule, so a connect string shared between a sender and a +query client fails identically on both sides. The Node port should match those +strings for the same reason — someone debugging a shared `ws::` string should +not get two different diagnoses from two clients. `X-QWP-Client-Id` follows Java's convention of `/` — Java 1.3.7 sends the constant `"java/1.0.2"`, which is deliberately **not** the @@ -805,6 +820,11 @@ fails immediately — and that both arrive *after* a successful handshake. ### 6.5.2 TLS — `tls_roots` does not port directly +**All three TLS keys require the `wss::` schema.** Supplying `tls_verify`, +`tls_roots` or `tls_roots_password` with plain `ws::` throws +"tls_verify/tls_roots/tls_roots_password require the wss:: schema" — they are not +silently ignored. `tls_roots_password` additionally requires `tls_roots`. + `tls_verify` maps cleanly: `on` → default verification, `unsafe_off` → `rejectUnauthorized: false`. Java additionally enforces that **a custom trust store may not be combined with disabled validation** (its constructor throws); @@ -1558,7 +1578,12 @@ are the only ones left. - `tls_roots` **cannot** be combined with `tls_verify=unsafe_off`. Java's message names both escapes — "remove tls_verify to use custom roots, or remove tls_roots to disable certificate validation" — and is worth copying verbatim. -- WebSocket requires at least one `host:port` pair in `addr` (1.2). +- `tls_roots_password` **requires** `tls_roots`. +- All three TLS keys **require `wss::`** — rejected under plain `ws::` (6.5.2). +- `username` **requires** `password` and vice versa; `token` **excludes** both + (6.5). +- WebSocket requires at least one `host:port` pair in `addr` (1.2); an empty + `addr` fails with "addr is missing". **Mode selection is implicit.** There is no `store_and_forward=on` key: **`sf_dir` present means disk mode, absent means memory mode.** That single fact drives From 7c11e57a448ed9a7f306d70e0c5ee285e4ebc28d Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 20:12:24 +0100 Subject: [PATCH 023/121] docs: bring multi-host failover into scope and add the addr grammar Scope correction from the user: multiple addresses are required for failover, so section 1.2's single-endpoint restriction is wrong and is replaced. Multi-host addressing and failover are now in scope. Every rotation-flavoured behaviour elsewhere in the spec is live rather than vestigial: RETRIABLE_OTHER's rotation, the FAILED_OVER and ALL_ENDPOINTS_UNREACHABLE events, the 421 role reject retried until a primary appears, the mid-stream cap change behind the snapshot-once rule, and the catch-up cap gap on a smaller-cap node. Adds the addr grammar from ConfigView.parseEntry: comma-separated, IPv6-aware, duplicates rejected on (host, port). A custom port on IPv6 requires brackets, since an unbracketed multi-colon entry is read as a bare IPv6 host on the default port. Java's "IPv6 addresses are not supported" throw is UDP-only and does not constrain WebSocket. Adds endpoint selection from QwpHostHealthTracker: rounds with pickNext and beginRound, priority as the lexicographic (state, zoneTier) tuple with state outranking zone, and the state order HEALTHY, UNKNOWN, TRANSIENT_REJECT, TRANSPORT_ERROR, TOPOLOGY_REJECT. Records a finding that narrows the ask: the ingest sender constructs the tracker with the single-argument form, which collapses every zone tier to SAME, so ingest selection is state-only and zone-aware ranking is genuinely an egress feature. zone and target therefore stay accept-and-ignore, and porting zone ranking into the sender would build something Java's sender does not have. Records that background drainers must use a private round cursor with health-only recording so they cannot consume the foreground round, and adds that to the risks. Stack grows to sixteen PRs with 9a and 9b. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 110 ++++++++++++++---- 1 file changed, 87 insertions(+), 23 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 1e58111..b076a56 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -22,9 +22,12 @@ Each of these is a separate future spec: - the query client (`QwpQueryClient`, result-batch decode, bind values); - the `QuestDB` facade and its sender/query pooling; -- multi-host HA failover (`failover_*` keys, roles and zones); +- **zone-aware and role-aware endpoint *ranking*** (`zone=`, `target=primary|replica`) + — see 1.2, this is an egress-side feature and the ingest sender does not use it; - the UDP sender. +Multi-host addressing and failover **are** in scope (1.2). + Several Java classes look ingest-relevant and are not — do not port them here: - `QwpServerInfo` / `QwpServerInfoDecoder` decode a `SERVER_INFO` frame sent by a @@ -40,24 +43,71 @@ Several Java classes look ingest-relevant and are not — do not port them here: frames between the producer and I/O threads, which the event loop makes unnecessary, and the rest are replaced wholesale by `Buffer`. Neither carries protocol semantics. -- `QwpHostHealthTracker` is multi-host (1.1, 1.2). +`QwpHostHealthTracker` **is** ported — see 1.2. + +### 1.2 Multi-host addressing and failover -### 1.2 Single endpoint — and what that degrades +`addr` is a **list**, and the sender walks it. Every rotation-flavoured behaviour +elsewhere in this spec — `RETRIABLE_OTHER`'s endpoint rotation (7.2), the +`FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` events (4.2), the `421` role reject +retried indefinitely until a primary appears (6.5.1), the mid-stream cap change +that forces the snapshot-once rule (5.1), and the catch-up cap gap on a +smaller-cap node (7.5) — is live rather than vestigial. -Because multi-host failover is out of scope, this stack connects to **one** -endpoint and reconnects to that same endpoint. Several behaviours ported from -Java are phrased in terms of endpoint rotation; their *shapes* are kept intact so -the later HA spec is additive rather than a rewrite, but their single-host -meaning must be stated or an implementer will either build HA by accident or -silently drop them. +#### `addr` grammar (`ConfigView.parseEntry`) -| Behaviour | Single-endpoint meaning | +Comma-separated, IPv6-aware, and **duplicates are rejected** — the key is +`(host, port)`, so the same host twice on different ports is fine. + +| Form | Meaning | |---|---| -| `RETRIABLE_OTHER` (7.2) | Keep the distinct policy and category, but with nothing to rotate to it behaves as `RETRIABLE` with the zero-progress pacer. Do not collapse the enum. | -| `FAILED_OVER`, `ALL_ENDPOINTS_UNREACHABLE` (4.2) | Defined but never emitted. `ENDPOINT_ATTEMPT_FAILED`, `CONNECTED`, `RECONNECTED`, `AUTH_FAILED` all still fire. | -| Cap changing mid-stream (5.1) | Still reachable — a reconnect to a restarted or upgraded server can advertise a different `X-QWP-Max-Batch-Size`. The snapshot-once rule stands on its own merits. | -| Catch-up cap gap (7.5) | Effectively unreachable single-host, but retained: it costs one counter and becomes live the moment HA lands. | -| `addr` | Parsed as a single `host:port`. Accept a comma-separated list syntactically if Java does, but use only the first entry, and say so rather than failing obscurely. | +| `host` | default port | +| `host:9000` | explicit port | +| `[::1]:9000` | bracketed IPv6 with port | +| `[::1]` | bracketed IPv6, default port | +| `::1` | **unbracketed multi-colon = bare IPv6, default port** | + +A custom port on IPv6 therefore **requires** brackets. Distinct errors exist for +a missing `]`, a non-`:` following `]`, an empty host, and a duplicate entry. +Default port for `ws`/`wss` is 9000 (3.5). + +Note Java's `resolveIPv4` throws "IPv6 addresses are not supported" — that is +**UDP-only** (it needs a raw IPv4 int) and does not constrain WebSocket. + +#### Endpoint selection — state-ranked, in rounds + +Port `QwpHostHealthTracker`. Selection is a **round**: `pickNext()` returns the +highest-priority endpoint not yet attempted this round; the caller advances with +`beginRound()`; a round can be exhausted. + +Priority is the lexicographic tuple `(state, zoneTier)`, with **state +outranking zone**, so a known-good cross-zone host is preferred over an untried +local one. Host states rank: + +`HEALTHY` → `UNKNOWN` → `TRANSIENT_REJECT` → `TRANSPORT_ERROR` → `TOPOLOGY_REJECT` + +**The ingest sender is zone-blind.** It constructs the tracker with the +single-argument form, which passes `clientZone=null, targetPrimary=false` and +collapses every host's zone tier to `SAME` — so ingest selection is +**state-only**. Zone tiers (`SAME` → `UNKNOWN` → `OTHER`) and `target=primary` +belong to the query client. This is why `zone` and `target` remain accept-and-ignore +in section 9 even though failover is in scope: porting zone ranking into the +sender would build something Java's sender does not have. + +#### Concurrency: drainers must not consume the shared round + +`pickNext` and `recordX` are individually synchronized but **not atomic as a +pair**, so the foreground connect walk must be serialized single-file. + +Background orphan drainers (8.4) must **not** consume or poison the shared +round. They take a **private round cursor** with a walker-local attempted set +(claim-at-pick, so concurrent cursors never race the pick→record pair) and record +**health-only** results — state updates flow into the shared ledger that orders +everyone's picks, but the shared round is untouched. Getting this wrong makes a +drainer silently steal endpoints from the foreground sender's round. + +A third reference implementation exists for this component: the tracker mirrors +the **.NET** client's `QwpHostHealthTracker`. ## 2. Normative sources — and which ones are traps @@ -328,7 +378,7 @@ Java exposes three separate surfaces; the Node port mirrors all three: | Java | Fires on | |---|---| | `SenderErrorHandler` | rejections — carries category, policy, `fromFsn`/`toFsn`, `quarantinedPath` | -| `SenderConnectionListener` | seven kinds: `CONNECTED`, `DISCONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` — two are never emitted single-endpoint (1.2) | +| `SenderConnectionListener` | seven kinds: `CONNECTED`, `DISCONNECTED`, `RECONNECTED`, `FAILED_OVER`, `ENDPOINT_ATTEMPT_FAILED`, `ALL_ENDPOINTS_UNREACHABLE`, `AUTH_FAILED` — all live, given multi-host (1.2) | | `SenderProgressHandler` | the ACK watermark advancing | They share one delivery contract that must survive the port: @@ -365,7 +415,8 @@ They share one delivery contract that must survive the port: A connection event is not just a kind. It carries the host and port, the **previous** host and port, an attempt number, a round number, a cause, and a timestamp — the attempt/round pair is what makes reconnect storms diagnosable, so -carry it even though single-endpoint (1.2) pins the round. `AUTH_FAILED`'s cause +carry it — with multi-host (1.2) the round number is what distinguishes one +sweep of the endpoint list from the next. `AUTH_FAILED`'s cause is the auth failure from the upgrade (6.5.1), which is how the terminal credential case reaches the listener before the producer-side throw. @@ -922,7 +973,7 @@ discards data without saying so. | Policy | Behaviour | |---|---| | `RETRIABLE` | recycle the connection, replay from `ackedFsn + 1`; handler delivery is informational | -| `RETRIABLE_OTHER` | same replay, but rotate endpoints rather than back off against the same node (single-endpoint behaviour: 1.2) | +| `RETRIABLE_OTHER` | same replay, but rotate to the next endpoint rather than back off against the same node (1.2) | | `TERMINAL` | latch; next producer call throws; bytes stay on disk | | `ABANDONED` | the rows are gone; nothing throws and the sender keeps running; bytes preserved at `quarantinedPath` | @@ -1400,7 +1451,7 @@ registry's classification verbatim. **`Side.COMMON` + `Side.INGRESS` — implemented by our sender:** -`addr` (single `host:port` in this stack — see 1.2), `username`, `password`, +`addr` (comma-separated host:port list — grammar in 1.2), `username`, `password`, `token`, `tls_verify`, `tls_roots`, `tls_roots_password`, `auth_timeout_ms`, `connect_timeout`, `auto_flush`, `auto_flush_bytes`, `auto_flush_interval`, `auto_flush_rows`, @@ -1421,7 +1472,12 @@ Plus two pre-existing Node-client keys with no Java counterpart, carried over so `ws::` behaves like the other Node protocols: `init_buf_size`, `max_buf_size`. **`Side.EGRESS` — accept-and-ignore.** These configure the query client, and a -shared connect string must not break the sender: `target`, `failover`, +shared connect string must not break the sender. This still holds with failover +in scope (1.2): the ingest sender always walks the `addr` list, ranked by host +state, with no on/off switch and no zone/role input — its retry budget comes from +the `reconnect_*` keys, which are `Side.INGRESS`. The `failover_*`, `target` and +`zone` keys tune the *query* client's selection, so the sender accepts and +ignores them: `target`, `failover`, `failover_max_attempts`, `failover_backoff_initial_ms`, `failover_backoff_max_ms`, `failover_max_duration_ms`, `max_batch_rows`, `initial_credit`, `buffer_pool_size`, `compression`, `compression_level`, `client_id`, `zone`. @@ -1636,8 +1692,9 @@ Four tiers, all four required. ## 11. PR stack -Fourteen stacked PRs, each independently reviewable and green. PRs 1–8 are the -wire; 9–13 are the reliability story; PR 3 is the first point at which a user +Sixteen stacked PRs, each independently reviewable and green. PRs 1–8 are the +wire; 9–13 are the reliability story, including multi-host failover at 9a/9b; +PR 3 is the first point at which a user could actually use the feature. | # | PR | Gate | @@ -1651,10 +1708,12 @@ could actually use the feature. | 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | | 8 | defer-commit + commit frame (5.1.1) + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | +| 9a | Multi-host `addr` grammar incl. IPv6 + duplicate rejection (1.2) | unit | +| 9b | Host health tracker: state ranking, rounds, `RETRIABLE_OTHER` rotation, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | | 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | | 11 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | | 12 | `.symbol-dict` persistence + delta replay after recovery | crash tests | -| 13 | Slot locks (both kinds), orphan scan, drainers, `DATA_LOSS`/`ABANDONED` | crash tests | +| 13 | Slot locks (both kinds), orphan scan, drainers with private round cursors (1.2), `DATA_LOSS`/`ABANDONED` | crash tests | | 14 | Docs, examples, README support matrix, 4.3.0 release | — | ## 12. Risks @@ -1712,6 +1771,11 @@ could actually use the feature. oversized one after a cancelled row or an empty flush. Java hit this; the golden vectors must include a commit frame emitted after `cancelRow` and after an empty flush. +- **A drainer consuming the foreground round.** Background drainers must use a + private round cursor and health-only recording (1.2). Sharing the round lets a + drainer silently steal endpoints from the foreground sender's sweep, which + presents as unexplained `ALL_ENDPOINTS_UNREACHABLE` under load rather than as + an obvious bug. - **Inverted upgrade-failure retry.** Treating `401`/`403` as retriable spins forever against a server that will never accept the credentials; treating `421` as terminal kills a sender during an ordinary failover window (6.5.1). From 536eb65c7d746eceb7e7eb915c17df097c8d4ea5 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 20:24:33 +0100 Subject: [PATCH 024/121] docs: point-by-point consistency fixes after the failover scope change Twenty-fourth pass, reading sections 1 to 4 and 10 in order. Eight defects, most of them introduced by the scope change itself. 1.1: the QwpHostHealthTracker line was glued to the preceding bullet list and rendered as part of it. 1.2: the selection rule led with Java's (state, zoneTier) tuple and only then said the ingest sender is zone-blind, leaving it ambiguous whether to implement zone tiers. It now leads with the state-only ranking the port needs and says plainly not to implement zone tiers, while asking that the ranking function stay shaped for a later addition. 2: the sources table listed two reference implementations while 1.2 cites a third; .NET is now listed, since Java's tracker javadoc says it mirrors it and that makes it the best cross-check for endpoint selection. 3.2: the module layout had no component for endpoint selection at all after failover came into scope; adds endpoints.ts and hostTracker.ts with a description. 3.2: subsection 3.2.1 sat between two bullets of the module-layout list, interrupting it. Moved after the list. 3.4: the runtime-model table had no row for the connect walk, which is now a real concurrency constraint -- a single in-flight connect, and drainers with their own cursors. 4: the flagship example used a single address, so the spec's headline snippet did not exercise the newly in-scope capability. It now shows a list including an IPv6 literal, and points at 4.3 since construction may or may not connect. 4: the new-surface block showed no callbacks despite 4.2 defining three; adds all three and a pointer to 4.3. 10: mock-server tier described only single-endpoint scenarios; adds the multi-endpoint matrix that 9b needs. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 65 +++++++++++++------ 1 file changed, 45 insertions(+), 20 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index b076a56..b570533 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -43,6 +43,7 @@ Several Java classes look ingest-relevant and are not — do not port them here: frames between the producer and I/O threads, which the event loop makes unnecessary, and the rest are replaced wholesale by `Buffer`. Neither carries protocol semantics. + `QwpHostHealthTracker` **is** ported — see 1.2. ### 1.2 Multi-host addressing and failover @@ -80,19 +81,22 @@ Port `QwpHostHealthTracker`. Selection is a **round**: `pickNext()` returns the highest-priority endpoint not yet attempted this round; the caller advances with `beginRound()`; a round can be exhausted. -Priority is the lexicographic tuple `(state, zoneTier)`, with **state -outranking zone**, so a known-good cross-zone host is preferred over an untried -local one. Host states rank: +**For the ingest sender, rank by host state alone:** `HEALTHY` → `UNKNOWN` → `TRANSIENT_REJECT` → `TRANSPORT_ERROR` → `TOPOLOGY_REJECT` -**The ingest sender is zone-blind.** It constructs the tracker with the -single-argument form, which passes `clientZone=null, targetPrimary=false` and -collapses every host's zone tier to `SAME` — so ingest selection is -**state-only**. Zone tiers (`SAME` → `UNKNOWN` → `OTHER`) and `target=primary` -belong to the query client. This is why `zone` and `target` remain accept-and-ignore -in section 9 even though failover is in scope: porting zone ranking into the -sender would build something Java's sender does not have. +Java's full priority is the lexicographic tuple `(state, zoneTier)` with state +outranking zone — but **the ingest sender is zone-blind**. It constructs the +tracker with the single-argument form, passing `clientZone=null, +targetPrimary=false`, which collapses every host's zone tier to `SAME` and +degenerates the tuple to state alone. Zone tiers (`SAME` → `UNKNOWN` → `OTHER`) +and `target=primary` are used only by the query client. + +So **do not implement zone tiers here.** That is also why `zone` and `target` +remain accept-and-ignore in section 9 even though failover is in scope: porting +zone ranking into the sender would build something Java's sender does not have. +Leave the ranking function shaped so a zone tier can be added later without +restructuring it. #### Concurrency: drainers must not consume the shared round @@ -123,6 +127,7 @@ machine and they do not agree with each other. | Status-code reference | `https://questdb.com/docs/connect/wire-protocols/qwp-ingress-websocket/` — the URL `WebSocketResponse` itself cites | | NACK policy rationale | `java-questdb-client/design/qwp-nack-policy-v2.md` — **rationale only**, see 2.1 | | Second reference implementation | `c-questdb-client` (Rust): `src/ws/`, `src/egress/`, `src/ingress/sender/qwp_ws*` | +| Third reference implementation | the **.NET** client — Java's `QwpHostHealthTracker` javadoc states it mirrors .NET's, so that is the best cross-check for endpoint selection (1.2) | **Stale — do not use:** `docs/qwp/{wire-ingress,sf-client,wire-egress,failover}.md` in the parent `questdb` repo. These were **deleted from master** by `d1c5b03415` @@ -184,6 +189,7 @@ src/qwp/ sf/ engine.ts ring.ts segment.ts manifest.ts ackWatermark.ts symbolDictFile.ts crc32c.ts slotLock.ts orphanScanner.ts drainer.ts + endpoints.ts hostTracker.ts sendLoop.ts transport.ts buffer.ts @@ -197,6 +203,17 @@ src/qwp/ fully implemented (see 3.2.1) — "no fragmentation" applies to data, not to the RFC's control obligations. +- **`protocol/`** — pure functions over `Buffer`. No I/O, no `async`. Directly + testable against golden vectors. +- **`sf/`** — store-and-forward. Ports `CursorSendEngine`, `SegmentRing`, + `MmapSegment`, `SegmentManager`, `SfManifest`, `AckWatermark`, + `PersistedSymbolDict`, `SlotLock`, `OrphanScanner`, `BackgroundDrainer`. +- **`endpoints.ts` / `hostTracker.ts`** — `addr` list parsing and the + state-ranked, round-based endpoint selection of 1.2. Port of + `QwpHostHealthTracker`. +- **`sendLoop.ts`** — publish → wire → ACK → trim. Port of + `CursorWebSocketSendLoop`. + #### 3.2.1 Control frames are not optional - **PING → PONG**, echoing the payload. A server that pings and gets no pong @@ -214,13 +231,6 @@ under backpressure must never have a control frame interleaved into the middle o its byte stream. Either write data frames as a single `socket.write()` call, or queue control frames behind the in-flight frame — never both writers into one partially-written frame. -- **`protocol/`** — pure functions over `Buffer`. No I/O, no `async`. Directly - testable against golden vectors. -- **`sf/`** — store-and-forward. Ports `CursorSendEngine`, `SegmentRing`, - `MmapSegment`, `SegmentManager`, `SfManifest`, `AckWatermark`, - `PersistedSymbolDict`, `SlotLock`, `OrphanScanner`, `BackgroundDrainer`. -- **`sendLoop.ts`** — publish → wire → ACK → trim. Port of - `CursorWebSocketSendLoop`. ### 3.3 Why hand-roll the WebSocket layer @@ -246,7 +256,8 @@ Java's three threads collapse onto the event loop: | producer thread | the caller's own code | | I/O send loop thread | an async task per connection | | segment manager thread | an async task using `fs.promises` (libuv threadpool) | -| background drainer threads | async tasks, each owning its own WebSocket | +| background drainer threads | async tasks, each owning its own WebSocket **and its own round cursor** (1.2) | +| lock-serialized foreground connect walk | a single in-flight connect promise; `pickNext`→`record` must not interleave (1.2) | No `worker_threads`, no native dependencies. Two Java primitives have no core Node equivalent and are replaced: @@ -311,7 +322,8 @@ Unchanged from the user's point of view — the protocol is a connect-string change: ```ts -const sender = Sender.fromConfig("ws::addr=localhost:9000;"); +// addr is a list; the sender walks it and fails over (1.2) +const sender = Sender.fromConfig("ws::addr=node1:9000,node2:9000,[::1]:9000;"); await sender.table("trades") .symbol("symbol", "ETH-USD") .floatColumn("price", 2615.54) @@ -326,8 +338,16 @@ await sender.flush(); // publish; does NOT wait for ACK const fsn = await sender.flushAndGetSequence(); // highest FSN published, or -1 const ok = await sender.drain(30_000); // flush + await ACK watermark sender.reset(); // discard buffered rows (see 4.1) + +// three separate callback surfaces (4.2) +sender.onError((e) => { /* e.category, e.policy, e.fromFsn, e.toFsn */ }); +sender.onConnectionEvent((e) => { /* e.kind, e.host, e.attempt, e.round */ }); +sender.onProgress((ackedFsn) => { /* watermark advanced */ }); ``` +Construction may or may not connect, depending on a **derived** default — see +4.3 before assuming either. + ### 4.1 `reset()` must also roll back the symbol watermark `reset()` discards every buffered row across **all** table buffers — but @@ -1672,7 +1692,12 @@ Four tiers, all four required. drives the whole error matrix on demand: each NACK status, malformed frames, mid-frame disconnect, slow-consumer backpressure, server-initiated close, and poison-detector escalation. A real QuestDB will not produce `INTERNAL_ERROR` - or a torn frame to order. Escalation needs **both** its conditions exercised + or a torn frame to order. It must also be startable as **several endpoints at + once**, to exercise 1.2: rotation on `RETRIABLE_OTHER`, a `421` role reject + that resolves when a primary appears, `FAILED_OVER` and + `ALL_ENDPOINTS_UNREACHABLE`, state ranking preferring a known-good host over + an untried one, and a drainer's private cursor not consuming the foreground + round. Escalation needs **both** its conditions exercised (7.4): a case that accrues 4 strikes *inside* the dwell window and asserts that escalation does **not** fire, alongside one that crosses both. 3. **Testcontainers integration.** Extends the existing From 5d88829143fd496a234d6bea49f5c4a61296d47b Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 20:32:34 +0100 Subject: [PATCH 025/121] docs: resolve the staging-buffer decision and renumber the PR stack Twenty-fifth pass, continuing the linear read from section 5. 5.3 told the implementer to "choose one and state it" between copy-on-append and porting the two-buffer swap, then recommended copy-on-append without deciding. A spec that defers its own fork leaves PR 3 with an open question, so copy-on-append is now the decision, with the two-buffer swap recorded as a rejected alternative and why. 5.3 also instructed that the 30s buffer-recycle timeout be dropped from 5.1 rather than left as dead prose -- and then 5.1 still listed it as a mid-split failure cause. That is exactly the dead prose 5.3 predicted. 5.1 now names the append deadline as the Node cause and says explicitly that Java's other cause has no analogue here. 5's data-flow diagram listed auto_flush_bytes as a trigger without qualification, contradicting 9.1, where it is off by default. Annotated. 11's table had sixteen rows numbered 1 to 14 with 9a and 9b wedged in, an artifact of inserting failover mid-stack. Renumbered 1 to 16 and corrected the phase description, plus the four cross-references that named PR numbers: the dictionary-mode note, the memory/disk ring note, and the publish-semantics risk, which spanned a now-shifted range. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 60 +++++++++++-------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index b570533..8776991 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -496,7 +496,8 @@ If the ring is at its `sf_max_total_bytes` cap, `flush()` awaits space for up to sender.table("t").symbol("s","x").doubleColumn("p",1.5).at(ts) -> QwpBuffer routes into a per-table TableBuffer (column-wise typed arrays; a column not set in a given row is marked null in that row's bitmap) - -> auto_flush_rows | auto_flush_bytes | auto_flush_interval, or flush() + -> auto_flush_rows | auto_flush_interval, or flush() + (auto_flush_bytes is a fourth trigger but is OFF by default -- see 9.1) -> frameEncoder.seal(): all dirty tables -> ONE frame, assigned an FSN (payload optionally zstd-compressed when negotiated) ...unless the encoded frame exceeds the server's cap, in which case @@ -549,7 +550,8 @@ is load-bearing rather than cosmetic: So this must be a real error class in the Node port, not a string check. **The split is deliberately not atomic across frames.** A publish failure at -frame `k > 1` (backpressure deadline, recycle timeout) leaves frames `1..k-1` on +frame `k > 1` — in Node, the `sf_append_deadline_millis` backpressure deadline +(5.3 removes Java's other cause, the buffer-recycle timeout) — leaves frames `1..k-1` on the ring as deferred-but-uncommitted. The error propagates past the reset-table-buffers step, so the source rows survive and the *next* flush re-emits the whole batch; the eventual commit then commits the already-published prefix @@ -607,7 +609,7 @@ The delta dictionary is not simply on or off: The mode is therefore a consequence of what durable state exists, not a user toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict -mode — which is what makes PR 6 shippable before PR 12. +mode — which is what makes PR 6 shippable before PR 14. ### 5.3 The staging buffer and its swap @@ -625,7 +627,8 @@ encoder and the SF ring, with a four-state lifecycle — 3. swap the *other* buffer in as active; 4. **if that buffer is still in use, wait for it to be recycled**, bounded at 30 s, throwing "Timeout waiting for buffer to be recycled" on expiry — this is - the buffer-recycle timeout 5.1 names as a mid-split failure cause; + the buffer-recycle timeout — which the Node port does **not** inherit, per the + decision below; 5. reset the newly active buffer; 6. mark the sealed buffer `SENDING` and hand it to the engine, which appends and returns once published. @@ -633,16 +636,21 @@ encoder and the SF ring, with a four-state lifecycle — The buffer stays pinned until the wire send completes, which is what the second buffer is for: the producer keeps filling while the previous batch is in flight. -**Node decision.** The pin exists because the buffer is read asynchronously after -handoff. In Node the same hazard appears at every `await` inside append, so -choose one and state it: either **copy on append** — the segment write takes -ownership of its own bytes, one staging buffer suffices, and the swap machinery -disappears — or **port the two-buffer swap** with an equivalent bounded wait. -Copy-on-append is recommended: Node must copy into a `Buffer` for the write -anyway, so the copy Java avoids is not actually avoidable here, and it removes -a timeout error path rather than reproducing it. If copy-on-append is chosen, -the 30 s recycle timeout referenced in 5.1 becomes unreachable and should be -dropped from that list rather than left as dead prose. +**Node decision: copy on append.** The pin exists because the buffer is read +asynchronously after handoff, and in Node that hazard recurs at every `await` +inside append. The segment write takes ownership of its own bytes, so **one** +staging buffer suffices and the swap machinery disappears entirely. + +This is the decision, not a recommendation: Node must copy into a `Buffer` for +the write regardless, so the copy Java's double-buffering avoids is not actually +avoidable here — we would pay it *and* carry a timeout error path. Consequently +the 30 s recycle wait has **no Node analogue** and must not appear as a failure +cause anywhere in this spec (5.1 has been corrected accordingly). + +*Rejected alternative:* port the two-buffer swap with an equivalent bounded wait. +It reproduces a timeout that cannot fire for any reason a Node implementation +would recognise, and buys nothing, since the copy it exists to avoid is +unavoidable. ## 6. Wire format @@ -1134,7 +1142,7 @@ Node README already documents for ILP ("each worker thread needs its own Sender instance"), so it introduces nothing new for users — but it does mean a slot directory is owned by exactly one `Sender` at a time, which 8.3 enforces. -Memory mode (PR 10) and disk mode (PR 11) share the ring **and the segment +Memory mode (PR 12) and disk mode (PR 13) share the ring **and the segment abstraction**: Java's `MmapSegment` has a `memoryBacked` flag selecting a malloc'd buffer instead of a file mapping, with the same cursor architecture. Port that flag rather than writing two segment types. @@ -1718,7 +1726,7 @@ Four tiers, all four required. ## 11. PR stack Sixteen stacked PRs, each independently reviewable and green. PRs 1–8 are the -wire; 9–13 are the reliability story, including multi-host failover at 9a/9b; +wire; 9–11 are error handling and failover; 12–15 are durability; 16 ships it. PR 3 is the first point at which a user could actually use the feature. @@ -1733,13 +1741,13 @@ could actually use the feature. | 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | | 8 | defer-commit + commit frame (5.1.1) + zstd (feature-detected) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | -| 9a | Multi-host `addr` grammar incl. IPv6 + duplicate rejection (1.2) | unit | -| 9b | Host health tracker: state ranking, rounds, `RETRIABLE_OTHER` rotation, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | -| 10 | Memory-mode ring — makes publish semantics safe | mock + e2e | -| 11 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | -| 12 | `.symbol-dict` persistence + delta replay after recovery | crash tests | -| 13 | Slot locks (both kinds), orphan scan, drainers with private round cursors (1.2), `DATA_LOSS`/`ABANDONED` | crash tests | -| 14 | Docs, examples, README support matrix, 4.3.0 release | — | +| 10 | Multi-host `addr` grammar incl. IPv6 + duplicate rejection (1.2) | unit | +| 11 | Host health tracker: state ranking, rounds, `RETRIABLE_OTHER` rotation, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | +| 12 | Memory-mode ring — makes publish semantics safe | mock + e2e | +| 13 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | +| 14 | `.symbol-dict` persistence + delta replay after recovery | crash tests | +| 15 | Slot locks (both kinds), orphan scan, drainers with private round cursors (1.2), `DATA_LOSS`/`ABANDONED` | crash tests | +| 16 | Docs, examples, README support matrix, 4.3.0 release | — | ## 12. Risks @@ -1760,10 +1768,10 @@ could actually use the feature. stream that trips the raw fallback, and columns of exactly 0, 1, 2 and 3 values — the sub-3 cases take a different path (6.3.1) and are where an off-by-one hides. -- **Publish-semantics `flush()` before PR 10.** Between PR 3 and PR 10 there is - no retention, so an unacked frame lost to a disconnect is lost. PRs 3–9 must +- **Publish-semantics `flush()` before PR 12.** Between PR 3 and PR 12 there is + no retention, so an unacked frame lost to a disconnect is lost. PRs 3–11 must document this in-tree and the feature must not be announced as - production-ready until PR 10 lands. + production-ready until PR 12 lands. - **Config-key ownership cannot be guessed.** `ConfigSchema` assigns every key a `Side`, and several ingest-sounding keys (`max_batch_rows`, `initial_credit`, `compression`, `client_id`) are `Side.EGRESS`. Port the registry as data with From 65b78b0553cc93af787aa000d78ff2cae7e15584 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 20:36:05 +0100 Subject: [PATCH 026/121] docs: remove ingest-side zstd, which is an egress-only feature Twenty-sixth pass, linear read of section 6 onward. Section 6.1 listed FLAG_ZSTD and section 6.2 never said which region it covers, which PR 8 could not have answered. Chasing that found the flag does not belong to the ingest path at all. The server-side constant is explicit: FLAG_ZSTD is "set only on RESULT_BATCH frames and only after the handshake negotiated zstd". Every reference in the Java client is on the decode side -- QwpResultBatchDecoder, QwpQueryClient, and a comment in WebSocketClient. The ingest encoder sets FLAG_GORILLA and ORs in FLAG_DELTA_SYMBOL_DICT and never sets FLAG_ZSTD. The negotiation is about the response direction too: the client sends X-QWP-Accept-Encoding to tell the server how to compress result batches, and the echoed X-QWP-Content-Encoding is parsed only so callers can observe the level applied. This corrects a decision taken on a false premise. The spec required zstd on ingest, feature-detected against Node's zstdCompress so the Node 20 floor could be kept. None of that is needed: the ingest sender performs no compression, PR 8 carries only defer-commit and the commit frame, and the version-floor question disappears. The compression and compression_level keys being Side.EGRESS is corroborating evidence rather than coincidence. Compression moves to the out-of-scope list alongside the other egress features. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 62 ++++++++++++++----- 1 file changed, 45 insertions(+), 17 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 8776991..7db5292 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -24,7 +24,10 @@ Each of these is a separate future spec: - the `QuestDB` facade and its sender/query pooling; - **zone-aware and role-aware endpoint *ranking*** (`zone=`, `target=primary|replica`) — see 1.2, this is an egress-side feature and the ingest sender does not use it; -- the UDP sender. +- the UDP sender; +- **payload compression** — `FLAG_ZSTD`, `X-QWP-Accept-Encoding`, + `compression` / `compression_level` are all egress-only (9.3); the ingest path + never compresses. Multi-host addressing and failover **are** in scope (1.2). @@ -499,7 +502,7 @@ sender.table("t").symbol("s","x").doubleColumn("p",1.5).at(ts) -> auto_flush_rows | auto_flush_interval, or flush() (auto_flush_bytes is a fourth trigger but is OFF by default -- see 9.1) -> frameEncoder.seal(): all dirty tables -> ONE frame, assigned an FSN - (payload optionally zstd-compressed when negotiated) + (no compression on the ingest path -- see 9.3) ...unless the encoded frame exceeds the server's cap, in which case it is split -- see 5.1 -> sf.append(frame) <-- flush() resolves here @@ -685,6 +688,9 @@ across frames. Flags: `DEFER_COMMIT 0x01`, `GORILLA 0x04`, `DELTA_SYMBOL_DICT 0x08`, `ZSTD 0x10`. +An ingest client sets at most the first three. **`ZSTD` is egress-only** — it +appears on `RESULT_BATCH` frames and is never set by a sender (9.3). + Both `GORILLA` and `DELTA_SYMBOL_DICT` are genuinely optional on the wire — `QwpMessageCursor` branches on `isGorillaEnabled()` / `isDeltaSymbolDictEnabled()` per message. Java's encoder always sets both, but the server accepts neither. @@ -1524,10 +1530,11 @@ why the ignore-lists must be explicit rather than a catch-all. Java implements this the same way and comments that "forward-compat is via the spec, not silent ignore". -**There is no `zstd` configuration key.** `zstd` is an enum *value* of the -egress-side `compression` key (`zstd` | `raw` | `auto`). Ingest-side zstd is -therefore purely a handshake negotiation (9.3) with no connect-string control — -do not invent a key for it. +**There is no `zstd` configuration key, and no ingest-side compression at all +(9.3).** `zstd` is an enum *value* of the egress-side `compression` key +(`zstd` | `raw` | `auto`), which configures how the server compresses **result +batches**. Do not invent an ingest compression key, and do not implement ingest +compression. ### 9.1 Defaults differ from ILP — do not inherit the ILP ones @@ -1674,16 +1681,37 @@ present means disk mode, absent means memory mode.** That single fact drives memory-vs-disk throughout section 8, including the mode-dependent `sf_max_total_bytes` default (9.1), and the spec never stated it. -### 9.3 zstd and the Node version floor - -`node:zlib`'s `zstdCompress` landed in **Node 22.15.0**. The client's documented -floor is Node 20 and CI runs `[20, 22, latest]`. A naive "require Node 22" rule -would still be wrong for 22.0–22.14. - -Therefore: **feature-detect**. Probe for `zstdCompressSync` at connect time. If -present, send `X-QWP-Content-Encoding: zstd` and set `FLAG_ZSTD`; if absent, -negotiate uncompressed. The floor stays at Node 20 and the CI matrix is -unchanged. +### 9.3 There is no ingest-side compression + +**Correction to an earlier decision.** This spec previously required zstd on the +ingest path, feature-detected against Node's `zstdCompress` (which landed in +22.15.0) so the Node 20 floor could be kept. That was built on a false premise. + +`FLAG_ZSTD` is **egress-only**. The server-side constant is explicit: *"Set only +on `RESULT_BATCH` frames and only after the handshake negotiated zstd."* Every +reference to it in the Java client is on the decode side — +`QwpResultBatchDecoder`, `QwpQueryClient`, and a comment in `WebSocketClient`. +The ingest encoder sets `FLAG_GORILLA` and ORs in `FLAG_DELTA_SYMBOL_DICT`, and +**never sets `FLAG_ZSTD`**. + +The negotiation is likewise about the *response* direction. The client sends +`X-QWP-Accept-Encoding` (e.g. `zstd;level=1,raw`) to tell the server how to +compress **result batches**; when the header is omitted the server ships them +uncompressed. The echoed `X-QWP-Content-Encoding` is parsed only so callers can +observe the level the server actually applied. + +Consequences: + +- the ingest sender performs **no compression at all**, and PR 8 carries only + defer-commit and the commit frame; +- the Node version floor question **evaporates** — nothing on this path needs + `zstdCompress`, so Node 20 is fine for reasons that have nothing to do with + feature detection; +- `compression` / `compression_level` are `Side.EGRESS` (section 9) precisely + because they configure this, which is corroborating evidence rather than a + coincidence; +- compression belongs to the query spec, alongside `X-QWP-Accept-Encoding` and + `X-QWP-Max-Batch-Rows`. ## 10. Testing @@ -1739,7 +1767,7 @@ could actually use the feature. | 5 | VARCHAR/BINARY/arrays/decimals/geohash/uuid/long256/char/ipv4 + their per-type rules (6.5.3) | golden + e2e | | 6 | Symbol dictionary: full-dict mode, then delta mode + `DICTIONARY_GAP` (5.2) | golden + e2e | | 7 | Gorilla timestamps (6.3.2) + int32-overflow raw fallback | golden + e2e | -| 8 | defer-commit + commit frame (5.1.1) + zstd (feature-detected) | e2e both on and off | +| 8 | defer-commit + commit frame (5.1.1) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | | 10 | Multi-host `addr` grammar incl. IPv6 + duplicate rejection (1.2) | unit | | 11 | Host health tracker: state ranking, rounds, `RETRIABLE_OTHER` rotation, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | From a14d239983c861cdf6090f1e973aec70a0fd8158 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 20:46:25 +0100 Subject: [PATCH 027/121] docs: correct what actually drives endpoint rotation Twenty-seventh pass, linear read of sections 7, 8 and 12. Section 1.2 claimed RETRIABLE_OTHER's endpoint rotation was made live by the failover scope change. Reading section 7 against it shows that is an over-claim: RETRIABLE_OTHER's only category is NOT_WRITABLE (0x0C), which 7.1 already records as reserved and not emitted by any current server. The policy is therefore still unreachable, and rotation is driven by something else entirely -- the tracker recording TRANSPORT_ERROR for a failed connect and TOPOLOGY_REJECT for a 421 role reject, both of which demote a host in the next pick. 1.2 now separates the two, saying plainly that connect-time failure drives rotation and that RETRIABLE_OTHER should still be mapped and implemented but not expected to fire, since today's servers signal the same condition with a reconnect-eligible close. 7.2's row carries the same caveat so the two sections read consistently, and the test matrix and PR 11 no longer name a policy that cannot be triggered -- a mock-server test written against RETRIABLE_OTHER would have needed the server to emit a reserved status byte. Sections 7.1, 7.3, 7.4 and 12 verified consistent: the category count of ten with seven wire-mapped matches Java's own framing, and none of the twenty risk entries referenced the now-removed ingest zstd. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 27 ++++++++++++------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 7db5292..6b5e3d1 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -51,12 +51,21 @@ Several Java classes look ingest-relevant and are not — do not port them here: ### 1.2 Multi-host addressing and failover -`addr` is a **list**, and the sender walks it. Every rotation-flavoured behaviour -elsewhere in this spec — `RETRIABLE_OTHER`'s endpoint rotation (7.2), the -`FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` events (4.2), the `421` role reject -retried indefinitely until a primary appears (6.5.1), the mid-stream cap change -that forces the snapshot-once rule (5.1), and the catch-up cap gap on a -smaller-cap node (7.5) — is live rather than vestigial. +`addr` is a **list**, and the sender walks it. Behaviours that were vestigial +under a single endpoint are now live: the `FAILED_OVER` / +`ALL_ENDPOINTS_UNREACHABLE` events (4.2), the `421` role reject retried +indefinitely until a primary appears (6.5.1), the mid-stream cap change that +forces the snapshot-once rule (5.1), and the catch-up cap gap on a smaller-cap +node (7.5). + +**What actually drives rotation is connect-time failure, not a NACK policy.** +The tracker records `TRANSPORT_ERROR` for a failed connect and `TOPOLOGY_REJECT` +for a `421` role reject, and those states demote a host in the next pick. The +`RETRIABLE_OTHER` policy (7.2) is the *mid-stream* rotation path, and its only +category — `NOT_WRITABLE` (0x0C) — is **reserved and not currently emitted by any +server** (7.1). So keep `RETRIABLE_OTHER` mapped and implemented, but do not +expect to reach it: today's servers signal the same condition with a +reconnect-eligible close, which routes through the transport path above. #### `addr` grammar (`ConfigView.parseEntry`) @@ -1007,7 +1016,7 @@ discards data without saying so. | Policy | Behaviour | |---|---| | `RETRIABLE` | recycle the connection, replay from `ackedFsn + 1`; handler delivery is informational | -| `RETRIABLE_OTHER` | same replay, but rotate to the next endpoint rather than back off against the same node (1.2) | +| `RETRIABLE_OTHER` | same replay, but rotate to the next endpoint rather than back off against the same node. **Unreachable today** — its only category `NOT_WRITABLE` is reserved (7.1); real rotation comes from connect-time failures (1.2) | | `TERMINAL` | latch; next producer call throws; bytes stay on disk | | `ABANDONED` | the rows are gone; nothing throws and the sender keeps running; bytes preserved at `quarantinedPath` | @@ -1729,7 +1738,7 @@ Four tiers, all four required. mid-frame disconnect, slow-consumer backpressure, server-initiated close, and poison-detector escalation. A real QuestDB will not produce `INTERNAL_ERROR` or a torn frame to order. It must also be startable as **several endpoints at - once**, to exercise 1.2: rotation on `RETRIABLE_OTHER`, a `421` role reject + once**, to exercise 1.2: rotation on a transport failure, a `421` role reject that resolves when a primary appears, `FAILED_OVER` and `ALL_ENDPOINTS_UNREACHABLE`, state ranking preferring a known-good host over an untried one, and a drainer's private cursor not consuming the foreground @@ -1770,7 +1779,7 @@ could actually use the feature. | 8 | defer-commit + commit frame (5.1.1) | e2e both on and off | | 9 | ACK/NACK matrix, `defaultPolicyFor`, reconnect, replay, dict catch-up (7.5), poison detector | mock server | | 10 | Multi-host `addr` grammar incl. IPv6 + duplicate rejection (1.2) | unit | -| 11 | Host health tracker: state ranking, rounds, `RETRIABLE_OTHER` rotation, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | +| 11 | Host health tracker: state ranking, rounds, rotation on transport error / `421`, `FAILED_OVER` / `ALL_ENDPOINTS_UNREACHABLE` | mock server, multi-endpoint | | 12 | Memory-mode ring — makes publish semantics safe | mock + e2e | | 13 | Disk segments (`SF01`), manifest, ack watermark, CRC32C, `fdatasync` | crash tests | | 14 | `.symbol-dict` persistence + delta replay after recovery | crash tests | From 8168a52904afc18aaa2afaccc7e635df43b3abe3 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:05:23 +0100 Subject: [PATCH 028/121] docs: complete the configuration defaults and note durability without recovery Twenty-eighth pass, reading section 9 straight through -- it had heavy targeted work across passes 18 to 22 but no linear read since it grew to five subsections. The defaults table was missing six entries, three of which change behaviour rather than tuning: drain_orphans off, request_durable_ack off, transaction off, plus sf_durability defaulting to memory and sf_dir unset meaning memory mode. drain_orphans defaulting to off is the significant one. Combined with sf_dir set, a crashed process's slot is written to disk, survives, and is never drained automatically -- nothing replays it until an operator enables the flag or another sender adopts the slot. The default is defensible, since draining opens background connections at startup, but a user who configures sf_dir expecting crash recovery gets durability without recovery. Recorded in 9.1 and added to the risks, flagged for the README. Two key names do not say what they do, so section 9 now says it: transaction is the defer-commit switch, and sf_dir is what selects disk mode. Resolves an apparent conflict between 6.5 and 9. client_id is Side.EGRESS and accept-and-ignore, which reads as though the sender omits X-QWP-Client-Id. It does not -- the sender always emits its own constant, and the key only lets a query client override its identifier. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 6b5e3d1..4551800 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -1508,6 +1508,10 @@ registry's classification verbatim. `sf_append_deadline_millis`, `sf_dir`, `sf_durability`, `sf_max_segment_bytes`, `sf_max_total_bytes`, `sf_sync_interval_millis`, `transaction`. +Two of those names do not say what they do: **`transaction`** is the +defer-commit switch (5.1.1, `FLAG_DEFER_COMMIT`), and **`sf_dir`** is what +selects disk mode — there is no `store_and_forward` key (9.2). + `user` and `pass` are **aliases** of `username` and `password`, registered via `alias()`; both spellings must resolve. @@ -1525,6 +1529,11 @@ ignores them: `target`, `failover`, `failover_max_duration_ms`, `max_batch_rows`, `initial_credit`, `buffer_pool_size`, `compression`, `compression_level`, `client_id`, `zone`. +`client_id` being egress-side does **not** mean the sender omits +`X-QWP-Client-Id`. The sender always sends its own constant (6.5); the +connect-string key only lets a *query* client override its identifier, so we +accept and ignore it while still emitting the header. + **`Side.POOL` — accept-and-ignore.** The facade applies these and "the two clients ignore" them, so we ignore them too rather than reject: `sender_pool_min`, `sender_pool_max`, `query_pool_min`, `query_pool_max`, `acquire_timeout_ms`, @@ -1570,6 +1579,11 @@ not have yet: | `max_background_drainers` | 4 | | `max_name_len` | 127 | | `sender_id` | `"default"` | +| `sf_durability` | `memory` (8.2 — **not** power-loss durable) | +| `sf_dir` | unset ⇒ **memory mode** (9.2) | +| `drain_orphans` | **off** — see below | +| `request_durable_ack` | off | +| `transaction` (defer commit) | off | | `durable_ack_keepalive_interval_millis` | 200 (≤ 0 disables) | | close shutdown await | 30,000 | | max quarantined copies per slot | 64 | @@ -1589,6 +1603,15 @@ not have yet: | catch-up packing limit when cap unadvertised | 64 KiB | | max catch-up cap-gap attempts (orphan drainer only) | 16 | +**Orphan draining is off by default**, and that combination deserves stating +plainly: with `sf_dir` set but `drain_orphans` unset, a crashed process's slot is +written to disk, survives, and is **never drained automatically**. Nothing +replays it until an operator enables the flag or another sender adopts the slot. +The default is defensible — draining opens background connections at startup — +but a user who configures `sf_dir` expecting crash recovery gets durability +without recovery unless they also opt in. This belongs in the README next to +`sf_dir`, not only here. + **Byte-based auto-flush is off by default on WebSocket.** The builder's WS default is `0`, which is exactly what `auto_flush_bytes=off` sets — so the trigger is rows and interval only. `QwpWebSocketSender.DEFAULT_AUTO_FLUSH_BYTES` @@ -1868,6 +1891,9 @@ could actually use the feature. rule (5.1) is the known instance; the same hazard applies anywhere the port turns one of Java's synchronous sections into an async one. Prefer locals captured at entry over field reads. +- **Durability without recovery.** `sf_dir` set and `drain_orphans` left at its + default of off means a crashed process's slot persists and is never replayed + (9.1). Users will read `sf_dir` as "crash recovery" and get only half of it. - **Permanent stalls that look like disk-full.** The `.symbol-dict` liveness floor (8.1.2) is the clearest example: enforce `sf_max_total_bytes` as a naive directory-byte sum and a producer can wedge forever, across restarts, From a064c072e6a05ff11153acb41676f43cc6b77de5 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:10:41 +0100 Subject: [PATCH 029/121] docs: verify the golden-vector harness is buildable and record its template Twenty-ninth pass, aimed at the one assumption in section 10 that had never been checked: that a Java-side emitter could drive QwpWebSocketEncoder standalone. Tier 1 is the linchpin of the wire-correctness strategy, so an unverified premise there was the largest remaining risk. It holds. QwpWebSocketEncoderTest already drives the encoder with no server, and every class the harness needs is public. Records the concrete shape, including two things that would otherwise cost the implementer a cycle: encode(buffer) is a one-call path for a single table, with the beginMessage / addTable / finishMessage sequence needed only for multi-table frames and for controlling the delta-dictionary bounds; and the buffer is native memory, so the harness must copy out and free rather than leak per fixture, as the existing tests do via assertMemoryLeak. The test also exposed an API-level fact section 6.2.1 had missed. Nullability is a per-column construction choice, not a per-value one: getOrCreateColumn takes useNullBitmap as a parameter and a column created without it cannot represent a null at all. The Node port needs the same decision point at a column's first write. Notes QwpWebSocketSenderMultiEndpointTest as the Java reference for the multi-endpoint test matrix that PR 11 needs. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 32 ++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 4551800..0806918 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -732,6 +732,12 @@ and no bitmap follows; non-zero means a bitmap of `ceil(rowCount/8)` bytes follows. Java writes `1`; a decoder must treat any non-zero value as "bitmap present". +Nullability is a **per-column construction choice**, not a per-value one: +`getOrCreateColumn(name, type, useNullBitmap)` takes it as a parameter, and a +column created without it cannot represent a null at all. The Node port needs the +same decision point — a column's first write determines whether it carries a +bitmap for the life of the batch. + Bitmap semantics (`QwpNullBitmap`): **bit `i` set means row `i` is NULL**, bit order **LSB-first within each byte**. Row 9 is therefore byte 1, bit 1. @@ -1756,12 +1762,36 @@ Four tiers, all four required. tests assert byte-for-byte equality. This catches endianness, varint, zig-zag, bit-packing and null-bitmap drift at the point of the mistake rather than as a mysterious server NACK, and the recorded SHA makes drift visible. + + **This is verified to be buildable, not assumed.** `QwpWebSocketEncoderTest` + already drives the encoder standalone with no server, and every class the + harness needs is public. The shape is: + + ```java + try (QwpWebSocketEncoder encoder = new QwpWebSocketEncoder(); + QwpTableBuffer buffer = new QwpTableBuffer("trades")) { + QwpTableBuffer.ColumnBuffer col = + buffer.getOrCreateColumn("x", TYPE_LONG, /*useNullBitmap*/ false); + col.addLong(1); + buffer.nextRow(); + int size = encoder.encode(buffer); // single-table convenience + // bytes: encoder.getBuffer().getBufferPtr(), length `size` + } + ``` + + Two practical notes. `encode(buffer)` is a one-call path for a single table; + the `beginMessage` / `addTable` / `finishMessage` sequence is only needed for + multi-table frames and for controlling the delta-dictionary bounds (5.1.1). + And the buffer is **native memory** — the existing tests wrap in + `assertMemoryLeak`, so the harness must copy out and free rather than leak + per fixture. 2. **TypeScript mock QWP server.** Performs the upgrade, decodes frames, and drives the whole error matrix on demand: each NACK status, malformed frames, mid-frame disconnect, slow-consumer backpressure, server-initiated close, and poison-detector escalation. A real QuestDB will not produce `INTERNAL_ERROR` or a torn frame to order. It must also be startable as **several endpoints at - once**, to exercise 1.2: rotation on a transport failure, a `421` role reject + once**, to exercise 1.2 — `QwpWebSocketSenderMultiEndpointTest` is the Java + reference for this — covering rotation on a transport failure, a `421` role reject that resolves when a primary appears, `FAILED_OVER` and `ALL_ENDPOINTS_UNREACHABLE`, state ranking preferring a known-good host over an untried one, and a drainer's private cursor not consuming the foreground From 6c12996b4299bbe2ee2dbda5fadeb638eb0294f9 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:22:47 +0100 Subject: [PATCH 030/121] docs: specify frame parsing, including inbound defragmentation Thirtieth pass, walking PR 1's framing as an implementer writing ws/frame.ts. Pass twelve applied this exercise to the handshake and TLS only; the frame codec had never been walked. Adds 3.2.2. The headline correction: "never fragmented" in 3.2 describes what we SEND, and reading it as an inbound rule produces a client that rejects valid traffic. Java maintains a dedicated fragment buffer and accumulates inbound continuation frames into it, doubling and capped at the maximum receive size, with an explicit error rather than silent truncation on overflow. A server response or an intermediary may fragment even though our data frames do not. Records the rest of what the file needs and the spec did not state: parsing is an incremental state machine resumed across reads, since TCP delivers arbitrary boundaries and one data event is not one frame; the receive buffer is 64 KiB by default and grows when the write position comes within 1 KiB of the end; control frames carry at most 125 payload bytes and are never fragmented; RSV bits must be zero as we negotiate no extensions; inbound frames are never masked and a masked one is a protocol error; and Java's strict mode rejecting non-minimal length encodings is off by default, so accept them on receive while always emitting minimal lengths. Disambiguates a receive-buffer figure in 7.5 that silently referred to the server's 128 KiB buffer while 3.2.2 now documents the client's 64 KiB one. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 46 ++++++++++++++++--- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 0806918..a36685c 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -244,6 +244,34 @@ its byte stream. Either write data frames as a single `socket.write()` call, or queue control frames behind the in-flight frame — never both writers into one partially-written frame. +#### 3.2.2 Frame parsing — what PR 1 actually needs + +"Never fragmented" (3.2) describes what we **send**. It does not describe what we +must **accept**, and reading it that way produces a client that fails on valid +traffic. + +- **Inbound fragmentation must be supported.** Java maintains a dedicated + fragment buffer and accumulates continuation frames into it, growing by + doubling and capped at the maximum receive size; exceeding that cap is an + error, not a silent truncation. A server response or an intermediary may + fragment even though our data frames never do. +- **Parsing is incremental.** TCP delivers arbitrary byte boundaries, so the + parser is a state machine — awaiting header, awaiting payload, complete, + error — resumed across reads. In Node that means accumulating across `data` + events and never assuming one event is one frame. +- **Receive buffer**: 64 KiB default, grown when the write position comes within + 1 KiB of the end, capped at a configured maximum. +- **Control frames** carry at most **125** payload bytes (RFC 6455) and must + never be fragmented. +- **RSV bits must be zero** — we negotiate no extensions, so any set RSV bit is a + protocol error. +- **Server→client frames are never masked** (RFC 6455); a masked inbound frame is + a protocol error. Only our outbound frames are masked (3.2.1). +- Java exposes a strict mode rejecting **non-minimal length encodings** and + leaves it **off** by default. Match that: accept a non-minimal length on + receive rather than failing a connection over it, but always *emit* minimal + lengths. + ### 3.3 Why hand-roll the WebSocket layer Both existing reference implementations hand-roll it, and the Rust client @@ -1114,11 +1142,12 @@ Chunking rules: - The catch-up is packed against the server's advertised `X-QWP-Max-Batch-Size`. - **"Not advertised" is not "unbounded."** If the server omits the header (older build, or a derived cap that collapsed to zero), pack against - `UNCAPPED_CATCHUP_PACKING_LIMIT = 64 KiB` — deliberately well below the 128 KiB - default receive buffer. The transport still closes anything larger than the - receive buffer with WS 1009, and a catch-up-only close is deliberately - non-terminal, so an unchunked catch-up would reconnect into the identical - oversized frame forever. + `UNCAPPED_CATCHUP_PACKING_LIMIT = 64 KiB` — deliberately well below the + **server's** 128 KiB default receive buffer (not to be confused with the + *client's* 64 KiB one in 3.2.2). The transport still closes anything larger + than the server's receive buffer with WS 1009, and a catch-up-only close is + deliberately non-terminal, so an unchunked catch-up would reconnect into the + identical oversized frame forever. - The packing limit bounds **multi-entry** packing only. A single oversized entry is measured against a separate, more generous limit, so an entry that already shipped inside a data frame is never reclassified as unsendable. @@ -1822,7 +1851,7 @@ could actually use the feature. | # | PR | Gate | |---|---|---| -| 1 | `ws/`: framing, masking, handshake, upgrade-failure classification (6.5.1), TLS mapping (6.5.2), net/tls socket | unit + mock server | +| 1 | `ws/`: incremental framing + inbound defragmentation (3.2.2), control frames (3.2.1), masking, handshake, upgrade-failure classification (6.5.1), TLS mapping (6.5.2), net/tls socket | unit + mock server | | 2 | `protocol/`: header, varint/zigzag, LONG/DOUBLE/TIMESTAMP/SYMBOL inline | golden vectors | | 3 | Sender wiring: `ws://` config (4 sites, 3.5), `QwpBuffer`/`QwpTransport`, byte + interval auto-flush, cap-split (5.1) | **testcontainers e2e green** | | 4 | Remaining scalar types, null bitmap, row lifecycle rules (6.5.3) | golden + e2e | @@ -1899,6 +1928,11 @@ could actually use the feature. drainer silently steal endpoints from the foreground sender's sweep, which presents as unexplained `ALL_ENDPOINTS_UNREACHABLE` under load rather than as an obvious bug. +- **Reading "never fragmented" as an inbound rule.** It describes what we send; + the client must still defragment inbound continuation frames (3.2.2). A parser + that rejects them fails on valid traffic, and only under whatever conditions + cause a peer or intermediary to fragment — so it passes local tests and breaks + in someone's deployment. - **Inverted upgrade-failure retry.** Treating `401`/`403` as retriable spins forever against a server that will never accept the credentials; treating `421` as terminal kills a sender during an ordinary failover window (6.5.1). From 098cc9ce5d1c3859c97c025d39fe3454946b46f8 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:27:10 +0100 Subject: [PATCH 031/121] docs: add ACK-to-FSN correlation and the delta-dict runtime fallback Thirty-first pass, walking the previously unwalked PRs as an implementer. Two findings, both correctness-critical and both entirely absent. PR 9. Section 6.6 showed an OK frame carrying seq:u64 and section 8 discussed FSNs, with no bridge between them -- so the natural reading is seq == fsn. It is not. seq is a CONNECTION-SCOPED wire sequence, counting frames sent on the current connection from 0, while FSNs are monotonic for the life of the log and survive reconnects. The client keeps nextWireSeq and fsnAtZero per connection and computes ackedFsn = fsnAtZero + seq, re-establishing fsnAtZero at the replay start point after each reconnect. Storing the raw seq as an FSN works until the first reconnect and then trims from near the start of the log, discarding unacknowledged data. Also records the clamp: never trust an ACK beyond what was sent, because a malformed or replayed response would otherwise force a trim of segments the new server never saw, and ignore an ACK arriving before any send. PR 6. Section 5.2 presented the two dictionary modes as chosen at startup from available durable state. There is also a one-way runtime fallback: if .symbol-dict proves unwritable mid-run, the sender degrades to full-dict mode permanently. Java's reasoning is that the side file can start failing appends while SF's own segments stay writable, because the segments are pre-allocated and the dictionary is the one thing still growing -- so a fixed mode turns a survivable condition into permanent ingestion loss. Notes this is the second defence against the same root cause as the 8.1.2 liveness floor, covering the filesystem refusing rather than the cap being hit. Also records that the delta baseline advances only once a frame is queued onto the ring, never at allocation time, which is what makes a failed publish safe. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index a36685c..c569bb8 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -651,6 +651,33 @@ The mode is therefore a consequence of what durable state exists, not a user toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict mode — which is what makes PR 6 shippable before PR 14. +**Delta mode must be able to fall back at runtime.** The transition is one-way, +delta → full-dict, permanent for the rest of the sender's life, and triggered by +the `.symbol-dict` side file proving unwritable mid-run. + +The reasoning is worth keeping verbatim in a comment. That file can start +failing appends — full disk, exhausted quota — while SF's own segments stay +writable, *because the segments are pre-allocated and the dictionary is the one +thing still growing*. If the mode were fixed at startup, every subsequent +`flush()` would throw forever, and a condition store-and-forward exists to +survive would become total, permanent ingestion loss. Full-dict frames need no +side file at all, so degrading costs wire size and keeps ingestion alive. + +On fallback: the delta baseline stops being consulted (it reports `-1`, the +empty-delta value from 5.1.1) and the write-ahead persist becomes a no-op. + +Note this is the *second* defence against the same root cause. §8.1.2's liveness +floor stops `.symbol-dict`'s monotonic growth from wedging the producer against +`sf_max_total_bytes`; this stops it from wedging the producer when the filesystem +itself refuses. Implement both — they cover different failures. + +**When the baseline advances matters.** It moves only after a frame carrying the +batch's symbols is **queued onto the ring**, never at symbol-allocation time, and +only ever forward — a batch that introduced no new symbols leaves it untouched. +That ordering is what makes a failed publish safe: ids allocated but never +shipped stay reclaimable (4.1), and no frame on the wire references an id the +persisted dictionary lacks. + ### 5.3 The staging buffer and its swap Sections 5.1 and 8.3.1 refer to "seal and swap the buffer" without saying what @@ -1022,6 +1049,42 @@ error : status:u8 | seq:u64 | errLen:u16 | utf8 `MAX_ERROR_MESSAGE_LENGTH` is 1024. +### 6.6.1 Correlating an ACK to a frame — `seq` is **not** an FSN + +This is the bridge between section 6's wire format and section 8's FSNs, and +assuming `seq == fsn` is the natural mistake. It works until the first +reconnect, then silently corrupts the trim watermark. + +The `seq` on an OK frame is a **connection-scoped wire sequence**: the count of +frames sent **on the current connection**, starting at 0. FSNs, by contrast, are +monotonic for the life of the store-and-forward log and survive reconnects. The +client therefore keeps two values per connection: + +- `nextWireSeq` — how many frames it has sent on this connection; +- `fsnAtZero` — the FSN that wire sequence 0 corresponds to. + +and translates every ACK as: + +``` +ackedFsn = fsnAtZero + seq +``` + +After a reconnect, `nextWireSeq` restarts at 0 and `fsnAtZero` is re-established +at the replay start point (`ackedFsn + 1`). An implementation that stores the raw +`seq` as an FSN will, on the first reconnect, trim from near the start of the log +and discard unacknowledged data. + +**Clamp before applying.** Never trust an ACK beyond what has actually been sent: + +``` +capped = min(seq, nextWireSeq - 1), floored at 0 +``` + +Log when clamping fires. Java's reasoning is specific — a malformed or replayed +server response would otherwise force a trim of segments the *new* server has +never seen. An ACK arriving before any send on the connection +(`nextWireSeq == 0`) is ignored outright. + ## 7. Error handling ### 7.1 Categories @@ -1923,6 +1986,14 @@ could actually use the feature. oversized one after a cancelled row or an empty flush. Java hit this; the golden vectors must include a commit frame emitted after `cancelRow` and after an empty flush. +- **Treating `seq` as an FSN.** The ACK sequence is connection-scoped and + restarts at 0 on every reconnect (6.6.1). Storing it as an FSN works until the + first reconnect, then trims from near the start of the log and discards + unacknowledged data. Tests must reconnect mid-stream and assert the trim + watermark did not move backwards. +- **A fixed dictionary mode.** If `.symbol-dict` becomes unwritable and there is + no delta→full-dict fallback (5.2), every later flush throws forever and a + survivable condition becomes permanent ingestion loss. - **A drainer consuming the foreground round.** Background drainers must use a private round cursor and health-only recording (1.2). Sharing the round lets a drainer silently steal endpoints from the foreground sender's sweep, which From 6b6f95a3a5f5de5508418709866016c7243dc02a Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:42:03 +0100 Subject: [PATCH 032/121] docs: define the FSN model, which eight other sections depended on Thirty-second pass, walking PRs 10 to 16. The dominant gap sits under 12 and 13: the spec used FSNs throughout -- publishedFsn, ackedFsn, replay from ackedFsn+1, fsnAtZero -- without ever saying where they come from. They are not a process-local counter. Every segment header carries baseSeq, the FSN of its first frame, and the ring computes nextSeq as the active segment's baseSeq plus its frame count, with publishedFsn one below that. A fresh ring starts at -1; a recovered ring continues where the previous process stopped. So FSNs persist across restarts and are unique for the life of the log, which is exactly why the connection-scoped wire seq needs translating through fsnAtZero rather than being used directly. The two findings are halves of one model. Also records what recovery must do beyond collecting files: hold sealed segments in baseSeq order, validate contiguity so each segment's baseSeq plus frame count meets the next segment's baseSeq and the head matches the manifest, and quarantine a segment with a negative baseSeq rather than admitting it as position zero -- the guard against a corrupt file whose own baseSeq is unreadable, which would otherwise silently renumber every frame after it. Renumbers 8.1.x. The new subsection was added as 8.1.0, repeating the non-monotonic pattern fixed in pass eight; headings are now 8.1.1 through 8.1.6 with all nine cross-references updated. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 59 +++++++++++++++---- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index c569bb8..7e655e4 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -613,7 +613,7 @@ baseline as **both** bounds so the range is `[baseline+1 .. baseline]`. This is the only shape that is unconditionally correct in both dictionary modes (5.2): - **Delta mode** — the commit path does *not* write-ahead-persist the dictionary - (8.1.5). Shipping a symbol here would put an id on the wire that a recovered + (8.1.6). Shipping a symbol here would put an id on the wire that a recovered slot cannot rebuild from `.symbol-dict`, diverging the producer's dictionary from the surviving frames and **silently misattributing reused ids after a crash**. @@ -645,7 +645,7 @@ The delta dictionary is not simply on or off: - **Delta mode** — each frame carries only ids above the last shipped id. Used in memory mode, and in disk mode *once the persisted `.symbol-dict` has opened*. Safe only because a reconnect re-registers via the catch-up frame (7.5) and - recovery reseeds from the persisted file (8.1.5). + recovery reseeds from the persisted file (8.1.6). The mode is therefore a consequence of what durable state exists, not a user toggle. A build that has delta encoding but no `.symbol-dict` must use full-dict @@ -666,7 +666,7 @@ side file at all, so degrading costs wire size and keeps ingestion alive. On fallback: the delta baseline stops being consulted (it reports `-1`, the empty-delta value from 5.1.1) and the write-ahead persist becomes a no-op. -Note this is the *second* defence against the same root cause. §8.1.2's liveness +Note this is the *second* defence against the same root cause. §8.1.3's liveness floor stops `.symbol-dict`'s monotonic growth from wedging the producer against `sf_max_total_bytes`; this stops it from wedging the producer when the filesystem itself refuses. Implement both — they cover different failures. @@ -1260,13 +1260,44 @@ abstraction**: Java's `MmapSegment` has a `memoryBacked` flag selecting a malloc'd buffer instead of a file mapping, with the same cursor architecture. Port that flag rather than writing two segment types. +### 8.1.1 Where FSNs come from + +The spec references FSNs throughout — `publishedFsn`, `ackedFsn`, replay from +`ackedFsn + 1`, `fsnAtZero` (6.6.1) — without ever saying where they originate. +They are **not** a counter the process initialises to zero. + +Every segment header carries `baseSeq`, the FSN of its **first** frame (8.1.5), +and a segment's frame count is derived by scanning it. The ring computes: + +``` +nextSeq = activeSegment.baseSeq + activeSegment.frameCount +publishedFsn = nextSeq - 1 +``` + +So a fresh ring starts at `nextSeq = 0`, `publishedFsn = -1`, and a **recovered +ring continues numbering where the previous process stopped**. FSNs therefore +persist across restarts and are unique for the life of the store-and-forward log +— which is precisely why the connection-scoped wire `seq` needs translating +through `fsnAtZero` (6.6.1) rather than being used directly. + +**Recovery must order and validate the chain**, not just collect files: + +- sealed segments are held in `baseSeq` order, oldest first, and sorted on open; +- contiguity is checked — each segment's `baseSeq + frameCount` must meet the + next segment's `baseSeq`, and the chain head must match the manifest's + recorded head; +- a segment with a **negative `baseSeq`** is excluded from the chain and + quarantined (8.4) rather than being treated as position zero. A corrupt file + whose own `baseSeq` is unreadable is the case this guards, and admitting one + would silently renumber every frame after it. + **Publish barrier.** Each segment carries an `appendCursor` (producer-only) and a `publishedCursor`. The consumer **must not read any byte at offset `>= publishedOffset()`**. That single rule is what makes the whole thing lock-free, and it is easy to lose in a port where `await` interleaves differently than Java's threads. -### 8.1.1 Hot-spare provisioning — the producer never creates a segment +### 8.1.2 Hot-spare provisioning — the producer never creates a segment `SegmentManager` is a background worker that keeps every registered ring supplied with a **pre-created hot-spare segment**, and trims segments once their @@ -1290,7 +1321,7 @@ already-existing spare; it never waits on file creation. Omitting hot spares does not fail a test; it just moves an `open`+`allocate` onto the producer at every rotation. Port it. -### 8.1.2 The `.symbol-dict` liveness-floor deadlock — do not reintroduce +### 8.1.3 The `.symbol-dict` liveness-floor deadlock — do not reintroduce `sf_max_total_bytes` must **not** be enforced as a naive sum of everything in the slot directory. Java guards this with @@ -1308,7 +1339,7 @@ the shortfall. The producer stalls **permanently, and across restarts**, while the disk-full warning points at a trim that cannot help. Guaranteeing the minimum working set is what turns that permanent deadlock into ordinary backpressure. -### 8.1.3 Two distinct append failures +### 8.1.4 Two distinct append failures `SegmentRing.appendOrFsn` has two sentinels and they need opposite handling: @@ -1320,7 +1351,7 @@ working set is what turns that permanent deadlock into ordinary backpressure. Treating `PAYLOAD_TOO_LARGE` as backpressure would burn the full append deadline before failing, and report a timeout instead of the real cause. -### 8.1.4 Segment file format (`MmapSegment`) +### 8.1.5 Segment file format (`MmapSegment`) ``` 24-byte header: @@ -1341,7 +1372,7 @@ mapping-plus-fd barrier; only the latter is a portable power-loss barrier, so the Node port implements the `syncPublished()` semantics (write + `fdatasync`) and does not reproduce the legacy path. -### 8.1.5 Persisted symbol dictionary — load-bearing, not an optimisation +### 8.1.6 Persisted symbol dictionary — load-bearing, not an optimisation `/.symbol-dict` (`PersistedSymbolDict`) is the component most easily missed, and omitting it makes delta-encoded recovery silently impossible. @@ -1449,7 +1480,7 @@ and reject them at construction. **Disk mode alone is not power-loss durability.** `sf_dir` selects disk mode (9.2), but durability still defaults to `memory` — files are written and never explicitly synced. Power-loss survival requires `sf_durability=periodic`, which -in turn requires `sf_dir`. Section 8.1.5's note that `.symbol-dict` is +in turn requires `sf_dir`. Section 8.1.6's note that `.symbol-dict` is "page-cache durable, not host-crash durable" is the same property, and it applies to the segments too under the default. @@ -1902,7 +1933,7 @@ Four tiers, all four required. duplicate must not fail the test. Plus the abandonment path: corrupt a slot, assert it is quarantined with `quarantinedPath` set, `DATA_LOSS` / `ABANDONED` is delivered, and the sender keeps running. Plus the liveness - floor (8.1.2): a slot whose side files alone approach `sf_max_total_bytes` + floor (8.1.3): a slot whose side files alone approach `sf_max_total_bytes` must still accept writes. ## 11. PR stack @@ -1986,6 +2017,12 @@ could actually use the feature. oversized one after a cancelled row or an empty flush. Java hit this; the golden vectors must include a commit frame emitted after `cancelRow` and after an empty flush. +- **FSNs restarting at zero on recovery.** They derive from the segment chain's + `baseSeq`, not a process-local counter (8.1.1), so a recovered ring must + continue the previous numbering. Reinitialising to zero makes recovered frames + collide with new ones and corrupts every watermark that depends on FSN + uniqueness. A segment with a negative `baseSeq` must be quarantined, not + treated as position zero. - **Treating `seq` as an FSN.** The ACK sequence is connection-scoped and restarts at 0 on every reconnect (6.6.1). Storing it as an FSN works until the first reconnect, then trims from near the start of the log and discards @@ -2030,7 +2067,7 @@ could actually use the feature. default of off means a crashed process's slot persists and is never replayed (9.1). Users will read `sf_dir` as "crash recovery" and get only half of it. - **Permanent stalls that look like disk-full.** The `.symbol-dict` liveness - floor (8.1.2) is the clearest example: enforce `sf_max_total_bytes` as a + floor (8.1.3) is the clearest example: enforce `sf_max_total_bytes` as a naive directory-byte sum and a producer can wedge forever, across restarts, while logging a trim warning that can never help. Crash tests must include a slot whose side files alone approach the cap. From 8c815ba7e2b8d6f74a4d1655c98f45770467bbf0 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:44:15 +0100 Subject: [PATCH 033/121] docs: specify torn-tail segment recovery Thirty-third pass, following the question the previous pass opened: 8.1.1 said frame count is "derived by scanning" without saying how a crash mid-append is detected. PR 13 could not have been written from that. The mechanism is more careful than a scan-until-garbage loop. Recovery walks frames through positioned reads and maps the file only after the scan validates, so a sparse or unbacked page cannot fault the process; the file length is checked before the scan, after it, and again after mapping, with a short read or size change aborting recovery. The tail is the first bad CRC or the first frame whose declared length overruns the file, and both cursors position at the start of that frame. Non-zero bytes after the last valid frame mean an attempted-but-failed write and are reported; a clean partial fill reports zero. The part most likely to be got wrong is that the residue is NOT destroyed during the scan. After a mid-file tear the suffix can still hold frames with valid CRCs -- potentially the only surviving copy of real payloads -- so sanitisation waits for the whole chain to validate, and the justification differs by role. A sealed segment's suffix is zeroed on proof, when frame accounting came out complete, and a tear that actually cost frames fails closed with every byte left for operator extraction. The resumed active segment's tail is zeroed by policy, because replay can never reach frames past the tear and leaving them risks resurrecting stale frames on a later reseal. Adds the matching crash tests, which need hand-built segment files rather than a real crash, and a risk entry: sanitising early destroys data invisibly, since the recovered ring still looks consistent. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-nodejs-client-design.md | 48 ++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md index 7e655e4..2f1d78d 100644 --- a/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md +++ b/docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md @@ -1372,6 +1372,42 @@ mapping-plus-fd barrier; only the latter is a portable power-loss barrier, so the Node port implements the `syncPublished()` semantics (write + `fdatasync`) and does not reproduce the legacy path. +**Recovering a segment: finding where valid frames end.** This is how §8.1.1's +"frame count is derived by scanning" actually works, and it is more careful than +a scan-until-garbage loop. + +- **Read, don't map, while scanning.** Recovery walks frames through *positioned + reads* and only maps the file once the scan has validated it. Mapping first + would let a sparse or unbacked page fault the process. The file length is + checked before the scan, after it, and again after mapping; a short read or a + size change aborts recovery as an operational failure. The caller must keep + concurrent writers off the file throughout — no mapping-based implementation + can stay safe against uncoordinated mutation after the final check. +- **The tail is the first bad CRC**, or the first frame whose declared length + runs past the file end. Both cursors position at the **start** of that frame, + so the segment resumes appending over it. +- **Distinguish a torn tail from a clean partial fill.** Non-zero bytes after + the last valid frame mean a write was attempted and failed — warn and report + the byte count. A writer that simply never wrote past the last valid frame + reports zero and logs nothing. +- **Do not destroy the residue during the scan.** After a *mid-file* tear the + suffix can still contain frames with valid CRCs — potentially the only + surviving copy of real payloads. Whether it may be zeroed is a chain-level + decision the segment cannot make alone, so sanitisation happens only after the + whole chain validates, and the justification differs by role: + - a **sealed** segment's suffix is zeroed on **proof** — frame accounting came + out complete, so the residue cannot hold a replayable frame. A tear that + actually cost frames **fails closed** instead, leaving every byte on disk for + operator extraction; + - the **resumed active** segment's tail is zeroed by **policy** — past a + mid-file tear it may hold valid-CRC frames of genuinely unacked payloads, but + replay can never reach them because the FSN sequence breaks at the tear, and + leaving them risks resurrecting stale frames on a later reseal. + +The asymmetry is the point: proof where proof is available, policy where it is +not, and fail-closed rather than silently discarding bytes that might be the only +copy. + ### 8.1.6 Persisted symbol dictionary — load-bearing, not an optimisation `/.symbol-dict` (`PersistedSymbolDict`) is the component most easily @@ -1932,7 +1968,12 @@ Four tiers, all four required. so the assertion is "every row present", not "every row once", and a duplicate must not fail the test. Plus the abandonment path: corrupt a slot, assert it is quarantined with `quarantinedPath` set, `DATA_LOSS` / - `ABANDONED` is delivered, and the sender keeps running. Plus the liveness + `ABANDONED` is delivered, and the sender keeps running. Plus **torn-tail + recovery** (8.1.5), which needs hand-built segment files rather than a real + crash: a truncated final frame, a frame whose declared length overruns the + file, a bad CRC mid-file with valid frames *after* it, and a clean partial + fill — asserting the first three recover to the last valid frame and that only + the clean fill reports zero torn bytes. Plus the liveness floor (8.1.3): a slot whose side files alone approach `sf_max_total_bytes` must still accept writes. @@ -2017,6 +2058,11 @@ could actually use the feature. oversized one after a cancelled row or an empty flush. Java hit this; the golden vectors must include a commit frame emitted after `cancelRow` and after an empty flush. +- **Zeroing a torn tail during the scan.** After a mid-file tear the residue can + hold valid-CRC frames that are the only surviving copy of real payloads + (8.1.5). Sanitising before the chain validates destroys them, and the loss is + invisible — the recovered ring looks consistent. Sanitisation must wait for + chain validation and fail closed when accounting comes out short. - **FSNs restarting at zero on recovery.** They derive from the segment chain's `baseSeq`, not a process-local counter (8.1.1), so a recovered ring must continue the previous numbering. Reinitialising to zero makes recovered frames From eba0ec7ae45367cb28f5bd103ce2f4d02de64cdc Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 22:00:37 +0100 Subject: [PATCH 034/121] docs: add implementation plan 1, the ws:// walking skeleton Covers spec PRs 1 to 3 in twelve TDD tasks, ending with a testcontainers test that proves a real QuestDB accepts our frames. Scoped to one of four plans rather than one plan for all sixteen PRs, since the stack spans three independent subsystems and each plan should produce working software on its own. Records a verification finding that simplifies the work: Sender.flush() is exactly buffer.toBufferNew() followed by transport.send(buf), so QwpBuffer can seal a whole frame in toBufferNew() and no existing interface needs widening in this plan. The spec's 3.1 claim that the drain half must be widened only starts biting at cap-splitting, which is Plan 2. Three defects found while checking the plan's own API assumptions against current main, fixed before the plan shipped: Sender.fromConfig is async and was not awaited; the integration test passed auto_flush=off, which spec 9.2 records as rejected for WebSocket; and it used a guessed testcontainers log-wait pattern where the existing integration test uses none. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-07-qwp-plan-1-walking-skeleton.md | 2008 +++++++++++++++++ 1 file changed, 2008 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-07-qwp-plan-1-walking-skeleton.md diff --git a/docs/superpowers/plans/2026-08-07-qwp-plan-1-walking-skeleton.md b/docs/superpowers/plans/2026-08-07-qwp-plan-1-walking-skeleton.md new file mode 100644 index 0000000..757ec44 --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-qwp-plan-1-walking-skeleton.md @@ -0,0 +1,2008 @@ +# QWP Plan 1 — Walking Skeleton (spec PRs 1–3) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship `ws::` ingest end-to-end in `@questdb/nodejs-client` — a QuestDB server accepts rows sent over QWP with LONG, DOUBLE, TIMESTAMP and SYMBOL columns, verified by SQL in a testcontainers test. + +**Architecture:** Four layers under `src/qwp/`, no upward dependencies. `ws/` is a hand-rolled RFC 6455 codec over `net`/`tls`. `protocol/` is pure `Buffer` functions with no I/O. `buffer.ts` implements the existing `SenderBuffer` interface by accumulating columnar data and sealing one QWP frame in `toBufferNew()`. `transport.ts` implements the existing `SenderTransport` interface. Because `Sender.flush()` is exactly `buffer.toBufferNew()` → `transport.send(buf)`, **no existing interface is widened in this plan.** + +**Tech Stack:** TypeScript, Node ≥ 20, `node:net`, `node:tls`, `node:crypto`, `node:buffer`. Tests: vitest. Integration: testcontainers. **No new runtime dependencies.** + +**Source of truth:** `docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md`. Section references below (e.g. 6.2.1) point into it. + +## Global Constraints + +- **No new runtime dependencies.** The package currently depends only on `undici`. Do not add `ws`, a CRC library, or a WebSocket framework (spec 3.3). +- **Node 20 floor.** No API newer than Node 20. There is no zstd on this path (spec 9.3). +- **All integers little-endian** on the wire (spec 6). +- **`varint` means unsigned LEB128**, never zig-zag unless a codec explicitly asks (spec 6.0). +- **`V` = `valueCount` = non-null row count.** Column payloads are compacted; never write placeholder slots for nulls (spec 6.2.1). +- **Options stay `undefined` until set.** Never pre-seed defaults with `??`; "unset" and "set to the default" must remain distinguishable (spec 9.1.2). +- Existing tests must stay green: `pnpm test`, `pnpm typecheck`, `pnpm eslint`. +- Out of scope in this plan, do not build: store-and-forward, FSN/ACK correlation, reconnect, failover, Gorilla, delta symbol dictionary, cap-splitting, remaining column types. + +## File Structure + +| File | Responsibility | +|---|---| +| `src/qwp/protocol/varint.ts` | LEB128 encode/decode, size calculation | +| `src/qwp/protocol/constants.ts` | Magic, version, flags, type codes, limits | +| `src/qwp/protocol/tableBuffer.ts` | Per-table columnar accumulation, type lock, null tracking | +| `src/qwp/protocol/frameEncoder.ts` | Header + table block + schema + column payloads | +| `src/qwp/ws/mask.ts` | Per-frame CSPRNG mask key, XOR | +| `src/qwp/ws/frame.ts` | RFC 6455 frame encode + incremental parse + defragmentation | +| `src/qwp/ws/handshake.ts` | Upgrade request, `Sec-WebSocket-Accept`, response classification | +| `src/qwp/ws/socket.ts` | net/tls connect, frame send/receive, control frames | +| `src/qwp/buffer.ts` | `QwpBuffer implements SenderBuffer` | +| `src/qwp/transport.ts` | `QwpTransport implements SenderTransport` | +| `src/options.ts` | **modify** — 4 protocol-branch sites | +| `src/buffer/index.ts` | **modify** — branch on protocol before protocol_version | +| `src/transport/index.ts` | **modify** — `case WS/WSS` | +| `src/index.ts` | **modify** — exports | + +--- + +### Task 1: LEB128 varint + +**Files:** +- Create: `src/qwp/protocol/varint.ts` +- Test: `test/qwp/varint.test.ts` + +**Interfaces:** +- Consumes: nothing. +- Produces: `writeVarint(buf: Buffer, offset: number, value: number): number` (returns new offset), `varintSize(value: number): number`, `readVarint(buf: Buffer, offset: number): { value: number; offset: number }`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/varint.test.ts +import { describe, it, expect } from "vitest"; +import { writeVarint, varintSize, readVarint } from "../../src/qwp/protocol/varint"; + +describe("varint (unsigned LEB128)", () => { + it("encodes single-byte values", () => { + const b = Buffer.alloc(4); + expect(writeVarint(b, 0, 0)).toBe(1); + expect(b[0]).toBe(0x00); + expect(writeVarint(b, 0, 127)).toBe(1); + expect(b[0]).toBe(0x7f); + }); + + it("encodes multi-byte values with the continuation bit", () => { + const b = Buffer.alloc(4); + const end = writeVarint(b, 0, 128); + expect(end).toBe(2); + expect(b[0]).toBe(0x80); + expect(b[1]).toBe(0x01); + }); + + it("round-trips a range of values", () => { + for (const v of [0, 1, 127, 128, 300, 16383, 16384, 1_000_000]) { + const b = Buffer.alloc(10); + const end = writeVarint(b, 0, v); + expect(end).toBe(varintSize(v)); + expect(readVarint(b, 0)).toEqual({ value: v, offset: end }); + } + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd /home/nick/repos/nodejs-questdb-client && npx vitest run test/qwp/varint.test.ts` +Expected: FAIL — cannot resolve `../../src/qwp/protocol/varint`. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/protocol/varint.ts +import { Buffer } from "node:buffer"; + +/** Unsigned LEB128. 7 data bits per byte; high bit set means another byte follows. */ +export function writeVarint(buf: Buffer, offset: number, value: number): number { + if (value < 0 || !Number.isInteger(value)) { + throw new Error(`varint requires a non-negative integer, got ${value}`); + } + let v = value; + let o = offset; + while (v >= 0x80) { + buf[o++] = (v & 0x7f) | 0x80; + v = Math.floor(v / 128); + } + buf[o++] = v; + return o; +} + +export function varintSize(value: number): number { + let v = value; + let n = 1; + while (v >= 0x80) { + v = Math.floor(v / 128); + n++; + } + return n; +} + +export function readVarint( + buf: Buffer, + offset: number, +): { value: number; offset: number } { + let value = 0; + let shift = 1; + let o = offset; + for (;;) { + if (o >= buf.length) throw new Error("incomplete varint"); + const b = buf[o++]; + value += (b & 0x7f) * shift; + if ((b & 0x80) === 0) break; + shift *= 128; + } + return { value, offset: o }; +} +``` + +Note: `Math.floor(v / 128)` rather than `>>> 7` — `>>>` truncates to 32 bits and row counts can exceed that. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/varint.test.ts` +Expected: PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/varint.ts test/qwp/varint.test.ts +git commit -m "feat(qwp): add unsigned LEB128 varint codec" +``` + +--- + +### Task 2: Protocol constants + +**Files:** +- Create: `src/qwp/protocol/constants.ts` +- Test: `test/qwp/constants.test.ts` + +**Interfaces:** +- Produces: `QWP_MAGIC`, `QWP_VERSION`, `HEADER_SIZE`, `FLAG_DEFER_COMMIT`, `FLAG_GORILLA`, `FLAG_DELTA_SYMBOL_DICT`, `TYPE_LONG`, `TYPE_DOUBLE`, `TYPE_SYMBOL`, `TYPE_TIMESTAMP`, `MAX_COLUMNS_PER_TABLE`, `MAX_NAME_LENGTH`, `WRITE_PATH`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/constants.test.ts +import { describe, it, expect } from "vitest"; +import { QWP_MAGIC, HEADER_SIZE, QWP_VERSION, TYPE_LONG, TYPE_SYMBOL } from "../../src/qwp/protocol/constants"; + +describe("QWP constants", () => { + it("magic reads as 0x31505751 little-endian", () => { + expect(QWP_MAGIC.toString("ascii")).toBe("QWP1"); + expect(QWP_MAGIC.readUInt32LE(0)).toBe(0x31505751); + }); + + it("pins header size and version", () => { + expect(HEADER_SIZE).toBe(12); + expect(QWP_VERSION).toBe(1); + }); + + it("pins the type codes this plan uses", () => { + expect(TYPE_LONG).toBe(0x05); + expect(TYPE_SYMBOL).toBe(0x09); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/constants.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/protocol/constants.ts +import { Buffer } from "node:buffer"; + +/** ASCII "QWP1"; reads as 0x31505751 when interpreted little-endian. */ +export const QWP_MAGIC = Buffer.from("QWP1", "ascii"); +export const QWP_VERSION = 1; +export const HEADER_SIZE = 12; + +export const FLAG_DEFER_COMMIT = 0x01; +export const FLAG_GORILLA = 0x04; +export const FLAG_DELTA_SYMBOL_DICT = 0x08; + +// Column type codes (spec 6.3). Only the four this plan encodes. +export const TYPE_DOUBLE = 0x07; +export const TYPE_SYMBOL = 0x09; +export const TYPE_TIMESTAMP = 0x0a; +export const TYPE_LONG = 0x05; + +// Limits mirrored from the server (spec 6.4). +export const MAX_COLUMNS_PER_TABLE = 2048; +export const MAX_NAME_LENGTH = 127; +export const MAX_ROWS_PER_TABLE = 1_000_000; + +export const WRITE_PATH = "/write/v4"; +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/constants.test.ts` +Expected: PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/constants.ts test/qwp/constants.test.ts +git commit -m "feat(qwp): add protocol constants" +``` + +--- + +### Task 3: Table buffer — columnar accumulation + +**Files:** +- Create: `src/qwp/protocol/tableBuffer.ts` +- Test: `test/qwp/tableBuffer.test.ts` + +**Interfaces:** +- Consumes: `constants.ts`. +- Produces: `class QwpTableBuffer` with `constructor(name: string)`, `getOrCreateColumn(name: string, type: number): ColumnBuffer | null`, `nextRow(): void`, `reset(): void`, `get rowCount(): number`, `get columns(): ColumnBuffer[]`, `get name(): string`. `interface ColumnBuffer { name: string; type: number; values: (number | bigint | string)[]; nulls: boolean[]; size: number; }` + +Implements spec 6.5.3: type locked on first sight, duplicate column in a row is first-value-wins, `nextRow()` back-fills nulls so all columns stay equal length. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/tableBuffer.test.ts +import { describe, it, expect } from "vitest"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { TYPE_LONG, TYPE_DOUBLE } from "../../src/qwp/protocol/constants"; + +describe("QwpTableBuffer", () => { + it("back-fills nulls so all columns stay equal length", () => { + const t = new QwpTableBuffer("trades"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.getOrCreateColumn("b", TYPE_DOUBLE)!.values.push(2.5); + t.nextRow(); + expect(t.rowCount).toBe(2); + for (const c of t.columns) expect(c.size).toBe(2); + // "a" is null in row 1, "b" is null in row 0 + expect(t.columns.find((c) => c.name === "a")!.nulls).toEqual([false, true]); + expect(t.columns.find((c) => c.name === "b")!.nulls).toEqual([true, false]); + }); + + it("locks a column's type on first sight", () => { + const t = new QwpTableBuffer("x"); + t.getOrCreateColumn("c", TYPE_LONG); + expect(() => t.getOrCreateColumn("c", TYPE_DOUBLE)).toThrow(/type mismatch/i); + }); + + it("ignores a duplicate column within one row (first value wins)", () => { + const t = new QwpTableBuffer("x"); + t.getOrCreateColumn("c", TYPE_LONG)!.values.push(1); + expect(t.getOrCreateColumn("c", TYPE_LONG)).toBeNull(); + }); + + it("rejects an empty column name", () => { + const t = new QwpTableBuffer("x"); + expect(() => t.getOrCreateColumn("", TYPE_LONG)).toThrow(/empty/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/tableBuffer.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/protocol/tableBuffer.ts +import { MAX_COLUMNS_PER_TABLE, MAX_NAME_LENGTH } from "./constants"; + +export interface ColumnBuffer { + name: string; + type: number; + /** Non-null values only — the wire is compacted (spec 6.2.1). */ + values: (number | bigint | string)[]; + /** One entry per row; true means NULL. */ + nulls: boolean[]; + /** Rows accounted for so far, including nulls. */ + size: number; +} + +export class QwpTableBuffer { + readonly name: string; + private readonly cols: ColumnBuffer[] = []; + private readonly byName = new Map(); + private rows = 0; + + constructor(name: string) { + if (!name) throw new Error("table name cannot be empty"); + if (Buffer.byteLength(name, "utf8") > MAX_NAME_LENGTH) { + throw new Error(`table name too long [maxLength=${MAX_NAME_LENGTH}]`); + } + this.name = name; + } + + get rowCount(): number { + return this.rows; + } + + get columns(): ColumnBuffer[] { + return this.cols; + } + + /** Returns null when the column already holds a value for the in-progress row. */ + getOrCreateColumn(name: string, type: number): ColumnBuffer | null { + if (!name) throw new Error("column name cannot be empty"); + const existing = this.byName.get(name); + if (existing) { + if (existing.type !== type) { + throw new Error( + `Column type mismatch for column '${name}': columnType=${existing.type}, sentType=${type}`, + ); + } + // Already has a value for this row -> first value wins, silently. + if (existing.size > this.rows) return null; + existing.nulls.push(false); + existing.size++; + return existing; + } + if (Buffer.byteLength(name, "utf8") > MAX_NAME_LENGTH) { + throw new Error(`column name too long [maxLength=${MAX_NAME_LENGTH}]`); + } + if (this.cols.length >= MAX_COLUMNS_PER_TABLE) { + throw new Error( + `column count exceeds maximum: ${this.cols.length + 1} (max ${MAX_COLUMNS_PER_TABLE})`, + ); + } + // Back-fill this column as null for every row already closed. + const col: ColumnBuffer = { + name, + type, + values: [], + nulls: new Array(this.rows).fill(true), + size: this.rows, + }; + col.nulls.push(false); + col.size++; + this.cols.push(col); + this.byName.set(name, col); + return col; + } + + /** Closes the row, back-filling a null into every column that was not set. */ + nextRow(): void { + this.rows++; + for (const c of this.cols) { + while (c.size < this.rows) { + c.nulls.push(true); + c.size++; + } + } + } + + reset(): void { + this.cols.length = 0; + this.byName.clear(); + this.rows = 0; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/tableBuffer.test.ts` +Expected: PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/tableBuffer.ts test/qwp/tableBuffer.test.ts +git commit -m "feat(qwp): add columnar table buffer with null back-fill" +``` + +--- + +### Task 4: Frame encoder — header, schema, four column types + +**Files:** +- Create: `src/qwp/protocol/frameEncoder.ts` +- Test: `test/qwp/frameEncoder.test.ts` + +**Interfaces:** +- Consumes: `constants.ts`, `varint.ts`, `tableBuffer.ts`. +- Produces: `encodeFrame(tables: QwpTableBuffer[]): Buffer`. + +Layout per spec 6.1/6.2: 12-byte header, then per table `[varint nameLen][utf8][varint rowCount][varint colCount]`, schema `[varint nameLen][utf8][typeCode]`, then per column `[nullHeader:u8]` + optional bitmap + **compacted** values. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/frameEncoder.test.ts +import { describe, it, expect } from "vitest"; +import { encodeFrame } from "../../src/qwp/protocol/frameEncoder"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { TYPE_LONG, HEADER_SIZE } from "../../src/qwp/protocol/constants"; + +describe("encodeFrame", () => { + it("writes a valid 12-byte header", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(7); + t.nextRow(); + const f = encodeFrame([t]); + expect(f.subarray(0, 4).toString("ascii")).toBe("QWP1"); + expect(f.readUInt8(4)).toBe(1); // version + expect(f.readUInt8(5)).toBe(0); // flags: none in this plan + expect(f.readUInt16LE(6)).toBe(1); // tableCount + expect(f.readUInt32LE(8)).toBe(f.length - HEADER_SIZE); // payloadLen excludes header + }); + + it("emits nullHeader 0 and compacted values when there are no nulls", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(2); + t.nextRow(); + const f = encodeFrame([t]); + // ...header, table name "t", rowCount 2, colCount 1, schema "a"+type, then column + // nullHeader is the byte immediately after the schema entry. + const idx = f.indexOf(TYPE_LONG, HEADER_SIZE); + expect(f.readUInt8(idx + 1)).toBe(0); // nullHeader = no nulls + expect(f.readBigInt64LE(idx + 2)).toBe(1n); + expect(f.readBigInt64LE(idx + 10)).toBe(2n); + }); + + it("emits nullHeader 1, an LSB-first bitmap, and only non-null values", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.nextRow(); // row 1: "a" not set -> null + const f = encodeFrame([t]); + const idx = f.indexOf(TYPE_LONG, HEADER_SIZE); + expect(f.readUInt8(idx + 1)).toBe(1); // bitmap present + expect(f.readUInt8(idx + 2)).toBe(0b00000010); // bit 1 set -> row 1 is NULL + expect(f.readBigInt64LE(idx + 3)).toBe(1n); // only ONE value, not two + expect(f.length).toBe(idx + 3 + 8); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/frameEncoder.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/protocol/frameEncoder.ts +import { Buffer } from "node:buffer"; +import { writeVarint, varintSize } from "./varint"; +import { QwpTableBuffer, ColumnBuffer } from "./tableBuffer"; +import { + HEADER_SIZE, + QWP_MAGIC, + QWP_VERSION, + TYPE_DOUBLE, + TYPE_LONG, + TYPE_SYMBOL, + TYPE_TIMESTAMP, +} from "./constants"; + +function utf8Size(s: string): number { + return Buffer.byteLength(s, "utf8"); +} + +/** varint length + utf8 bytes (spec 6.0 "string"). */ +function writeString(buf: Buffer, offset: number, s: string): number { + const n = utf8Size(s); + let o = writeVarint(buf, offset, n); + buf.write(s, o, "utf8"); + return o + n; +} + +function stringSize(s: string): number { + const n = utf8Size(s); + return varintSize(n) + n; +} + +function columnPayloadSize(col: ColumnBuffer, rowCount: number): number { + const nullCount = col.nulls.filter(Boolean).length; + let n = 1; // nullHeader + if (nullCount > 0) n += Math.ceil(rowCount / 8); + const v = col.values.length; + switch (col.type) { + case TYPE_LONG: + case TYPE_DOUBLE: + case TYPE_TIMESTAMP: + return n + v * 8; + case TYPE_SYMBOL: { + // Inline dictionary: varint dictSize, entries, then a varint index per value. + const dict = [...new Set(col.values as string[])]; + n += varintSize(dict.length); + for (const s of dict) n += stringSize(s); + for (const s of col.values as string[]) n += varintSize(dict.indexOf(s)); + return n; + } + default: + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); + } +} + +function writeColumn( + buf: Buffer, + offset: number, + col: ColumnBuffer, + rowCount: number, +): number { + let o = offset; + const nullCount = col.nulls.filter(Boolean).length; + if (nullCount > 0) { + buf[o++] = 1; + const bytes = Math.ceil(rowCount / 8); + buf.fill(0, o, o + bytes); + for (let i = 0; i < rowCount; i++) { + // bit i set means row i is NULL, LSB-first within each byte (spec 6.2.1) + if (col.nulls[i]) buf[o + (i >>> 3)] |= 1 << (i & 7); + } + o += bytes; + } else { + buf[o++] = 0; + } + + switch (col.type) { + case TYPE_LONG: + case TYPE_TIMESTAMP: + for (const v of col.values) { + buf.writeBigInt64LE(BigInt(v as number | bigint), o); + o += 8; + } + return o; + case TYPE_DOUBLE: + for (const v of col.values) { + buf.writeDoubleLE(v as number, o); + o += 8; + } + return o; + case TYPE_SYMBOL: { + const dict = [...new Set(col.values as string[])]; + o = writeVarint(buf, o, dict.length); + for (const s of dict) o = writeString(buf, o, s); + for (const s of col.values as string[]) o = writeVarint(buf, o, dict.indexOf(s)); + return o; + } + default: + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); + } +} + +function tableSize(t: QwpTableBuffer): number { + let n = stringSize(t.name) + varintSize(t.rowCount) + varintSize(t.columns.length); + for (const c of t.columns) n += stringSize(c.name) + 1; + for (const c of t.columns) n += columnPayloadSize(c, t.rowCount); + return n; +} + +/** Encodes one QWP v1 message. No flags are set in this plan (spec 6.1). */ +export function encodeFrame(tables: QwpTableBuffer[]): Buffer { + const payloadLen = tables.reduce((a, t) => a + tableSize(t), 0); + const buf = Buffer.allocUnsafe(HEADER_SIZE + payloadLen); + + QWP_MAGIC.copy(buf, 0); + buf.writeUInt8(QWP_VERSION, 4); + buf.writeUInt8(0, 5); // flags + buf.writeUInt16LE(tables.length, 6); + buf.writeUInt32LE(payloadLen, 8); + + let o = HEADER_SIZE; + for (const t of tables) { + o = writeString(buf, o, t.name); + o = writeVarint(buf, o, t.rowCount); + o = writeVarint(buf, o, t.columns.length); + for (const c of t.columns) { + o = writeString(buf, o, c.name); + buf.writeUInt8(c.type, o++); + } + for (const c of t.columns) o = writeColumn(buf, o, c, t.rowCount); + } + if (o !== buf.length) { + throw new Error(`frame size mismatch: wrote ${o}, sized ${buf.length}`); + } + return buf; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/frameEncoder.test.ts` +Expected: PASS, 3 tests. The `o !== buf.length` assertion catches any size/write divergence immediately. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/frameEncoder.ts test/qwp/frameEncoder.test.ts +git commit -m "feat(qwp): encode QWP v1 frames with null bitmaps and compacted values" +``` + +--- + +### Task 5: WebSocket masking + +**Files:** +- Create: `src/qwp/ws/mask.ts` +- Test: `test/qwp/ws.mask.test.ts` + +**Interfaces:** +- Produces: `newMaskKey(): Buffer` (4 bytes from the OS CSPRNG), `applyMask(payload: Buffer, key: Buffer): void` (in place). + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/ws.mask.test.ts +import { describe, it, expect } from "vitest"; +import { newMaskKey, applyMask } from "../../src/qwp/ws/mask"; + +describe("ws masking", () => { + it("produces a fresh 4-byte key per call", () => { + const a = newMaskKey(); + const b = newMaskKey(); + expect(a.length).toBe(4); + // Not a strong randomness test; catches a constant/seeded-once key. + const keys = new Set([a.toString("hex"), b.toString("hex")]); + for (let i = 0; i < 20; i++) keys.add(newMaskKey().toString("hex")); + expect(keys.size).toBeGreaterThan(1); + }); + + it("is its own inverse", () => { + const key = Buffer.from([1, 2, 3, 4]); + const original = Buffer.from("hello websocket", "utf8"); + const payload = Buffer.from(original); + applyMask(payload, key); + expect(payload.equals(original)).toBe(false); + applyMask(payload, key); + expect(payload.equals(original)).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/ws.mask.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/ws/mask.ts +import { Buffer } from "node:buffer"; +import { randomFillSync } from "node:crypto"; + +/** RFC 6455 §10.3 requires a fresh, unpredictable key per frame. */ +export function newMaskKey(): Buffer { + return randomFillSync(Buffer.allocUnsafe(4)); +} + +export function applyMask(payload: Buffer, key: Buffer): void { + for (let i = 0; i < payload.length; i++) { + payload[i] ^= key[i & 3]; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ws.mask.test.ts` +Expected: PASS, 2 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/ws/mask.ts test/qwp/ws.mask.test.ts +git commit -m "feat(qwp): add per-frame websocket masking" +``` + +--- + +### Task 6: WebSocket frame codec + +**Files:** +- Create: `src/qwp/ws/frame.ts` +- Test: `test/qwp/ws.frame.test.ts` + +**Interfaces:** +- Consumes: `mask.ts`. +- Produces: `OPCODE = { CONT: 0x0, TEXT: 0x1, BINARY: 0x2, CLOSE: 0x8, PING: 0x9, PONG: 0xa }`, `encodeClientFrame(opcode: number, payload: Buffer): Buffer`, `class FrameParser` with `push(chunk: Buffer): void` and `next(): { opcode: number; payload: Buffer } | null`. + +`FrameParser` implements spec 3.2.2: incremental across chunks, **defragments inbound continuation frames**, rejects masked inbound frames, rejects non-zero RSV, caps control payloads at 125. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/ws.frame.test.ts +import { describe, it, expect } from "vitest"; +import { encodeClientFrame, FrameParser, OPCODE } from "../../src/qwp/ws/frame"; + +/** Server->client frames are never masked (RFC 6455). */ +function serverFrame(opcode: number, payload: Buffer, fin = true): Buffer { + const head: number[] = [(fin ? 0x80 : 0) | opcode]; + if (payload.length < 126) head.push(payload.length); + else if (payload.length < 65536) head.push(126, payload.length >>> 8, payload.length & 0xff); + else throw new Error("test helper: use a small payload"); + return Buffer.concat([Buffer.from(head), payload]); +} + +describe("ws frame codec", () => { + it("encodes a masked client binary frame", () => { + const f = encodeClientFrame(OPCODE.BINARY, Buffer.from([1, 2, 3])); + expect(f[0]).toBe(0x82); // FIN + binary + expect(f[1] & 0x80).toBe(0x80); // mask bit set + expect(f[1] & 0x7f).toBe(3); + expect(f.length).toBe(2 + 4 + 3); + }); + + it("uses the 64-bit length form above 65535", () => { + const f = encodeClientFrame(OPCODE.BINARY, Buffer.alloc(70000)); + expect(f[1] & 0x7f).toBe(127); + expect(Number(f.readBigUInt64BE(2))).toBe(70000); + }); + + it("parses a frame split across chunks", () => { + const whole = serverFrame(OPCODE.BINARY, Buffer.from("abcd")); + const p = new FrameParser(); + p.push(whole.subarray(0, 3)); + expect(p.next()).toBeNull(); + p.push(whole.subarray(3)); + expect(p.next()!.payload.toString()).toBe("abcd"); + }); + + it("defragments continuation frames", () => { + const p = new FrameParser(); + p.push(serverFrame(OPCODE.BINARY, Buffer.from("ab"), false)); + expect(p.next()).toBeNull(); + p.push(serverFrame(OPCODE.CONT, Buffer.from("cd"), true)); + const msg = p.next()!; + expect(msg.opcode).toBe(OPCODE.BINARY); + expect(msg.payload.toString()).toBe("abcd"); + }); + + it("rejects a masked inbound frame", () => { + const f = serverFrame(OPCODE.BINARY, Buffer.from("x")); + f[1] |= 0x80; // claim masked + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/masked/i); + }); + + it("rejects non-zero RSV bits", () => { + const f = serverFrame(OPCODE.BINARY, Buffer.from("x")); + f[0] |= 0x40; + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/rsv/i); + }); + + it("rejects an oversized control frame", () => { + const f = serverFrame(OPCODE.PING, Buffer.alloc(126)); + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/control frame/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/ws.frame.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/ws/frame.ts +import { Buffer } from "node:buffer"; +import { newMaskKey, applyMask } from "./mask"; + +export const OPCODE = { + CONT: 0x0, + TEXT: 0x1, + BINARY: 0x2, + CLOSE: 0x8, + PING: 0x9, + PONG: 0xa, +} as const; + +const MAX_CONTROL_PAYLOAD = 125; + +/** Client->server frames are always FIN=1 and always masked (spec 3.2.1). */ +export function encodeClientFrame(opcode: number, payload: Buffer): Buffer { + const len = payload.length; + let headerLen = 2; + if (len >= 65536) headerLen += 8; + else if (len >= 126) headerLen += 2; + + const out = Buffer.allocUnsafe(headerLen + 4 + len); + out[0] = 0x80 | opcode; + if (len < 126) { + out[1] = 0x80 | len; + } else if (len < 65536) { + out[1] = 0x80 | 126; + out.writeUInt16BE(len, 2); + } else { + out[1] = 0x80 | 127; + out.writeBigUInt64BE(BigInt(len), 2); + } + const key = newMaskKey(); + key.copy(out, headerLen); + payload.copy(out, headerLen + 4); + applyMask(out.subarray(headerLen + 4), key); + return out; +} + +export class FrameParser { + private buf: Buffer = Buffer.alloc(0); + private fragOpcode = -1; + private frags: Buffer[] = []; + + push(chunk: Buffer): void { + this.buf = this.buf.length === 0 ? chunk : Buffer.concat([this.buf, chunk]); + } + + /** Returns the next complete message, or null when more bytes are needed. */ + next(): { opcode: number; payload: Buffer } | null { + for (;;) { + if (this.buf.length < 2) return null; + const b0 = this.buf[0]; + const b1 = this.buf[1]; + + if ((b0 & 0x70) !== 0) throw new Error("websocket: non-zero RSV bits"); + if ((b1 & 0x80) !== 0) throw new Error("websocket: inbound frame must not be masked"); + + const fin = (b0 & 0x80) !== 0; + const opcode = b0 & 0x0f; + const isControl = (opcode & 0x08) !== 0; + + let len = b1 & 0x7f; + let offset = 2; + if (len === 126) { + if (this.buf.length < 4) return null; + len = this.buf.readUInt16BE(2); + offset = 4; + } else if (len === 127) { + if (this.buf.length < 10) return null; + const big = this.buf.readBigUInt64BE(2); + if (big > BigInt(Number.MAX_SAFE_INTEGER)) throw new Error("websocket: frame too large"); + len = Number(big); + offset = 10; + } + + if (isControl) { + if (len > MAX_CONTROL_PAYLOAD) { + throw new Error(`websocket: control frame payload exceeds ${MAX_CONTROL_PAYLOAD}`); + } + if (!fin) throw new Error("websocket: control frame must not be fragmented"); + } + + if (this.buf.length < offset + len) return null; + const payload = Buffer.from(this.buf.subarray(offset, offset + len)); + this.buf = this.buf.subarray(offset + len); + + // Control frames are never fragmented and interleave freely. + if (isControl) return { opcode, payload }; + + if (opcode === OPCODE.CONT) { + if (this.fragOpcode === -1) throw new Error("websocket: continuation without start"); + this.frags.push(payload); + if (!fin) continue; + const full = Buffer.concat(this.frags); + const op = this.fragOpcode; + this.frags = []; + this.fragOpcode = -1; + return { opcode: op, payload: full }; + } + + if (!fin) { + this.fragOpcode = opcode; + this.frags = [payload]; + continue; + } + return { opcode, payload }; + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ws.frame.test.ts` +Expected: PASS, 7 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/ws/frame.ts test/qwp/ws.frame.test.ts +git commit -m "feat(qwp): add RFC 6455 frame codec with inbound defragmentation" +``` + +--- + +### Task 7: Handshake — request, accept validation, failure classification + +**Files:** +- Create: `src/qwp/ws/handshake.ts` +- Test: `test/qwp/ws.handshake.test.ts` + +**Interfaces:** +- Consumes: `constants.ts`. +- Produces: `buildUpgradeRequest(opts): { request: Buffer; key: string }`, `computeAccept(key: string): string`, `parseUpgradeResponse(raw: Buffer): UpgradeResult`, `class QwpUpgradeError extends Error { status: number; kind: "role-reject" | "auth" | "other" }`. + +Implements spec 6.5 and 6.5.1: `421`+`X-QuestDB-Role` is a **retriable** role reject; `401`/`403` is a **terminal** auth failure; everything else (including `404`) is unclassified. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/ws.handshake.test.ts +import { describe, it, expect } from "vitest"; +import { + buildUpgradeRequest, + computeAccept, + parseUpgradeResponse, + QwpUpgradeError, +} from "../../src/qwp/ws/handshake"; + +describe("qwp handshake", () => { + it("computes Sec-WebSocket-Accept per RFC 6455", () => { + // The canonical example from RFC 6455 §1.3. + expect(computeAccept("dGhlIHNhbXBsZSBub25jZQ==")).toBe("s3pPLMBiTxaQ9kYGzzhZRbK+xOo="); + }); + + it("builds an upgrade request with the QWP headers", () => { + const { request } = buildUpgradeRequest({ host: "h", port: 9000, clientId: "nodejs/1.0.0" }); + const s = request.toString("ascii"); + expect(s).toMatch(/^GET \/write\/v4 HTTP\/1\.1\r\n/); + expect(s).toMatch(/\r\nUpgrade: websocket\r\n/); + expect(s).toMatch(/\r\nSec-WebSocket-Version: 13\r\n/); + expect(s).toMatch(/\r\nX-QWP-Max-Version: 1\r\n/); + expect(s).toMatch(/\r\nX-QWP-Client-Id: nodejs\/1\.0\.0\r\n/); + expect(s.endsWith("\r\n\r\n")).toBe(true); + }); + + it("classifies 421 with a role header as a retriable role reject", () => { + const raw = Buffer.from( + "HTTP/1.1 421 Misdirected Request\r\nX-QuestDB-Role: replica\r\n\r\n", + "ascii", + ); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect(e).toBeInstanceOf(QwpUpgradeError); + expect((e as QwpUpgradeError).kind).toBe("role-reject"); + expect((e as QwpUpgradeError).retriable).toBe(true); + } + }); + + it("classifies 401 as a terminal auth failure", () => { + const raw = Buffer.from("HTTP/1.1 401 Unauthorized\r\n\r\n", "ascii"); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect((e as QwpUpgradeError).kind).toBe("auth"); + expect((e as QwpUpgradeError).retriable).toBe(false); + } + }); + + it("leaves 404 unclassified", () => { + const raw = Buffer.from("HTTP/1.1 404 Not Found\r\n\r\n", "ascii"); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect((e as QwpUpgradeError).kind).toBe("other"); + } + }); + + it("returns negotiated headers on 101", () => { + const raw = Buffer.from( + "HTTP/1.1 101 Switching Protocols\r\n" + + "Upgrade: websocket\r\nConnection: Upgrade\r\n" + + "Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=\r\n" + + "X-QWP-Version: 1\r\nX-QWP-Max-Batch-Size: 1048576\r\n\r\n", + "ascii", + ); + const r = parseUpgradeResponse(raw); + expect(r.accept).toBe("s3pPLMBiTxaQ9kYGzzhZRbK+xOo="); + expect(r.qwpVersion).toBe(1); + expect(r.maxBatchSize).toBe(1048576); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/ws.handshake.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/ws/handshake.ts +import { Buffer } from "node:buffer"; +import { createHash, randomBytes } from "node:crypto"; +import { WRITE_PATH } from "../protocol/constants"; + +const WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + +export type UpgradeFailureKind = "role-reject" | "auth" | "other"; + +export class QwpUpgradeError extends Error { + readonly status: number; + readonly kind: UpgradeFailureKind; + /** 421 role rejects retry indefinitely; auth failures never do (spec 6.5.1). */ + readonly retriable: boolean; + readonly role?: string; + + constructor(status: number, kind: UpgradeFailureKind, message: string, role?: string) { + super(message); + this.name = "QwpUpgradeError"; + this.status = status; + this.kind = kind; + this.retriable = kind === "role-reject"; + this.role = role; + } +} + +export interface UpgradeResult { + accept: string; + qwpVersion?: number; + maxBatchSize?: number; + role?: string; + /** Bytes already received after the header terminator. */ + leftover: Buffer; +} + +export function computeAccept(key: string): string { + return createHash("sha1").update(key + WS_GUID, "ascii").digest("base64"); +} + +export function buildUpgradeRequest(opts: { + host: string; + port: number; + clientId: string; + authorization?: string; +}): { request: Buffer; key: string } { + const key = randomBytes(16).toString("base64"); + const lines = [ + `GET ${WRITE_PATH} HTTP/1.1`, + `Host: ${opts.host}:${opts.port}`, + "Connection: Upgrade", + "Upgrade: websocket", + "Sec-WebSocket-Version: 13", + `Sec-WebSocket-Key: ${key}`, + "X-QWP-Max-Version: 1", + `X-QWP-Client-Id: ${opts.clientId}`, + ]; + if (opts.authorization) lines.push(`Authorization: ${opts.authorization}`); + return { request: Buffer.from(lines.join("\r\n") + "\r\n\r\n", "ascii"), key }; +} + +export function parseUpgradeResponse(raw: Buffer): UpgradeResult { + const end = raw.indexOf("\r\n\r\n"); + if (end < 0) throw new Error("incomplete HTTP upgrade response"); + const head = raw.subarray(0, end).toString("ascii"); + const leftover = Buffer.from(raw.subarray(end + 4)); + + const [statusLine, ...headerLines] = head.split("\r\n"); + const status = Number.parseInt(statusLine.split(" ")[1], 10); + + const headers = new Map(); + for (const line of headerLines) { + const i = line.indexOf(":"); + if (i > 0) headers.set(line.slice(0, i).trim().toLowerCase(), line.slice(i + 1).trim()); + } + + if (status !== 101) { + const role = headers.get("x-questdb-role"); + if (status === 421 && role) { + throw new QwpUpgradeError(status, "role-reject", `node cannot accept writes [role=${role}]`, role); + } + if (status === 401 || status === 403) { + throw new QwpUpgradeError(status, "auth", `authentication failed [status=${status}]`); + } + throw new QwpUpgradeError(status, "other", `websocket upgrade failed [status=${status}]`); + } + + const accept = headers.get("sec-websocket-accept"); + if (!accept) throw new Error("upgrade response missing Sec-WebSocket-Accept"); + + const version = headers.get("x-qwp-version"); + const cap = headers.get("x-qwp-max-batch-size"); + return { + accept, + qwpVersion: version ? Number.parseInt(version, 10) : undefined, + maxBatchSize: cap ? Number.parseInt(cap, 10) : undefined, + role: headers.get("x-questdb-role"), + leftover, + }; +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ws.handshake.test.ts` +Expected: PASS, 6 tests. The `computeAccept` test uses RFC 6455's own worked example, so it validates against the standard rather than against our own output. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/ws/handshake.ts test/qwp/ws.handshake.test.ts +git commit -m "feat(qwp): add websocket handshake and upgrade-failure classification" +``` + +--- + +### Task 8: WebSocket socket — connect and send + +**Files:** +- Create: `src/qwp/ws/socket.ts` +- Test: `test/qwp/ws.socket.test.ts` + +**Interfaces:** +- Consumes: `frame.ts`, `handshake.ts`. +- Produces: `class QwpWebSocket` with `static connect(opts): Promise`, `sendBinary(payload: Buffer): Promise`, `close(): Promise`, `get maxBatchSize(): number | undefined`. + +Handles PING→PONG and CLOSE→echo (spec 3.2.1). Writes each data frame in a **single** `socket.write()` so a control frame can never interleave mid-frame. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/ws.socket.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { createServer, Server } from "node:net"; +import { createHash } from "node:crypto"; +import { QwpWebSocket } from "../../src/qwp/ws/socket"; +import { FrameParser, encodeClientFrame, OPCODE } from "../../src/qwp/ws/frame"; + +let server: Server | undefined; +afterEach(() => server?.close()); + +/** Minimal QWP-ish websocket server: completes the upgrade, echoes nothing. */ +function startServer(onBinary: (b: Buffer) => void): Promise { + return new Promise((resolve) => { + server = createServer((sock) => { + let handshaken = false; + const parser = new FrameParser(); + sock.on("data", (chunk) => { + if (!handshaken) { + const text = chunk.toString("ascii"); + const key = /Sec-WebSocket-Key: (.+)\r\n/.exec(text)![1]; + const accept = createHash("sha1") + .update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11", "ascii") + .digest("base64"); + sock.write( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n" + + `Connection: Upgrade\r\nSec-WebSocket-Accept: ${accept}\r\n` + + "X-QWP-Version: 1\r\nX-QWP-Max-Batch-Size: 1048576\r\n\r\n", + ); + handshaken = true; + return; + } + parser.push(chunk); + for (let m = parser.next(); m; m = parser.next()) { + if (m.opcode === OPCODE.BINARY) onBinary(m.payload); + if (m.opcode === OPCODE.PING) sock.write(encodeClientFrame(OPCODE.PONG, m.payload)); + } + }); + }); + server.listen(0, "127.0.0.1", () => resolve((server!.address() as any).port)); + }); +} + +describe("QwpWebSocket", () => { + it("connects, negotiates, and sends a binary frame", async () => { + const received: Buffer[] = []; + const port = await startServer((b) => received.push(b)); + const ws = await QwpWebSocket.connect({ + host: "127.0.0.1", + port, + tls: false, + clientId: "nodejs/1.0.0", + }); + expect(ws.maxBatchSize).toBe(1048576); + await ws.sendBinary(Buffer.from("payload")); + await new Promise((r) => setTimeout(r, 50)); + expect(received.length).toBe(1); + expect(received[0].toString()).toBe("payload"); + await ws.close(); + }); + + it("rejects a bad Sec-WebSocket-Accept", async () => { + server = createServer((sock) => { + sock.on("data", () => + sock.write( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n" + + "Connection: Upgrade\r\nSec-WebSocket-Accept: wrong\r\n\r\n", + ), + ); + }); + const port: number = await new Promise((r) => + server!.listen(0, "127.0.0.1", () => r((server!.address() as any).port)), + ); + await expect( + QwpWebSocket.connect({ host: "127.0.0.1", port, tls: false, clientId: "x" }), + ).rejects.toThrow(/accept/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/ws.socket.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/ws/socket.ts +import { Buffer } from "node:buffer"; +import { connect as netConnect, Socket } from "node:net"; +import { connect as tlsConnect } from "node:tls"; +import { encodeClientFrame, FrameParser, OPCODE } from "./frame"; +import { buildUpgradeRequest, computeAccept, parseUpgradeResponse } from "./handshake"; + +export interface QwpWebSocketOptions { + host: string; + port: number; + tls: boolean; + clientId: string; + authorization?: string; + rejectUnauthorized?: boolean; + ca?: Buffer | Buffer[]; +} + +export class QwpWebSocket { + private readonly socket: Socket; + private readonly parser = new FrameParser(); + private closed = false; + readonly maxBatchSize?: number; + + private constructor(socket: Socket, maxBatchSize?: number) { + this.socket = socket; + this.maxBatchSize = maxBatchSize; + this.socket.on("data", (chunk: Buffer) => this.onData(chunk)); + } + + static connect(opts: QwpWebSocketOptions): Promise { + return new Promise((resolve, reject) => { + const socket: Socket = opts.tls + ? tlsConnect({ + host: opts.host, + port: opts.port, + rejectUnauthorized: opts.rejectUnauthorized !== false, + ca: opts.ca, + }) + : netConnect({ host: opts.host, port: opts.port }); + + const onError = (e: Error) => reject(e); + socket.once("error", onError); + + socket.once(opts.tls ? "secureConnect" : "connect", () => { + const { request, key } = buildUpgradeRequest(opts); + socket.write(request); + + let acc = Buffer.alloc(0); + const onHeaderData = (chunk: Buffer) => { + acc = Buffer.concat([acc, chunk]); + if (acc.indexOf("\r\n\r\n") < 0) return; + socket.off("data", onHeaderData); + socket.off("error", onError); + try { + const res = parseUpgradeResponse(acc); + if (res.accept !== computeAccept(key)) { + throw new Error("websocket: Sec-WebSocket-Accept mismatch"); + } + const ws = new QwpWebSocket(socket, res.maxBatchSize); + if (res.leftover.length > 0) ws.onData(res.leftover); + resolve(ws); + } catch (e) { + socket.destroy(); + reject(e); + } + }; + socket.on("data", onHeaderData); + }); + }); + } + + private onData(chunk: Buffer): void { + this.parser.push(chunk); + for (let m = this.parser.next(); m; m = this.parser.next()) { + switch (m.opcode) { + case OPCODE.PING: + this.socket.write(encodeClientFrame(OPCODE.PONG, m.payload)); + break; + case OPCODE.CLOSE: + // RFC 6455 §5.5.1: echo the close before tearing down. + if (!this.closed) { + this.closed = true; + this.socket.write(encodeClientFrame(OPCODE.CLOSE, m.payload)); + this.socket.end(); + } + break; + default: + // Response frames are decoded in a later plan (ACK handling). + break; + } + } + } + + /** One write per frame, so a control frame can never interleave mid-frame. */ + sendBinary(payload: Buffer): Promise { + return new Promise((resolve, reject) => { + if (this.closed) return reject(new Error("websocket is closed")); + const frame = encodeClientFrame(OPCODE.BINARY, payload); + this.socket.write(frame, (err) => (err ? reject(err) : resolve())); + }); + } + + close(): Promise { + return new Promise((resolve) => { + if (this.closed) return resolve(); + this.closed = true; + this.socket.write(encodeClientFrame(OPCODE.CLOSE, Buffer.alloc(0))); + this.socket.end(() => resolve()); + }); + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ws.socket.test.ts` +Expected: PASS, 2 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/ws/socket.ts test/qwp/ws.socket.test.ts +git commit -m "feat(qwp): add websocket transport socket with control-frame handling" +``` + +--- + +### Task 9: `QwpBuffer` implementing `SenderBuffer` + +**Files:** +- Create: `src/qwp/buffer.ts` +- Test: `test/qwp/buffer.test.ts` + +**Interfaces:** +- Consumes: `tableBuffer.ts`, `frameEncoder.ts`, `constants.ts`. +- Produces: `class QwpBuffer implements SenderBuffer`. + +Unsupported column types throw explicitly rather than silently producing wrong bytes. `toBufferNew()` seals one frame across all dirty tables. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/buffer.test.ts +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../src/qwp/buffer"; +import { HEADER_SIZE } from "../../src/qwp/protocol/constants"; + +describe("QwpBuffer", () => { + it("seals a frame containing the buffered rows", () => { + const b = new QwpBuffer(); + b.table("trades").symbol("sym", "ETH").floatColumn("price", 1.5); + b.at(1000n, "us"); + const f = b.toBufferNew()!; + expect(f.subarray(0, 4).toString("ascii")).toBe("QWP1"); + expect(f.readUInt16LE(6)).toBe(1); // one table + expect(f.length).toBeGreaterThan(HEADER_SIZE); + }); + + it("returns null when nothing is buffered", () => { + expect(new QwpBuffer().toBufferNew()).toBeNull(); + }); + + it("accumulates multiple tables into one frame", () => { + const b = new QwpBuffer(); + b.table("a").intColumn("x", 1); + b.at(1n, "us"); + b.table("b").intColumn("y", 2); + b.at(2n, "us"); + expect(b.toBufferNew()!.readUInt16LE(6)).toBe(2); + }); + + it("throws for column types this plan does not encode", () => { + const b = new QwpBuffer(); + b.table("t"); + expect(() => b.booleanColumn("flag", true)).toThrow(/not supported/i); + }); + + it("clears state after sealing", () => { + const b = new QwpBuffer(); + b.table("t").intColumn("x", 1); + b.at(1n, "us"); + b.toBufferNew(); + expect(b.toBufferNew()).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/buffer.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/buffer.ts +import { Buffer } from "node:buffer"; +import { SenderBuffer } from "../buffer"; +import { TimestampUnit } from "../utils"; +import { QwpTableBuffer } from "./protocol/tableBuffer"; +import { encodeFrame } from "./protocol/frameEncoder"; +import { TYPE_DOUBLE, TYPE_LONG, TYPE_SYMBOL, TYPE_TIMESTAMP } from "./protocol/constants"; + +function toMicros(value: number | bigint, unit: TimestampUnit): bigint { + const v = typeof value === "bigint" ? value : BigInt(Math.trunc(value)); + switch (unit) { + case "ns": + return v / 1000n; + case "ms": + return v * 1000n; + default: + return v; + } +} + +function unsupported(what: string): never { + throw new Error(`${what} is not supported by the QWP buffer yet`); +} + +export class QwpBuffer implements SenderBuffer { + private tables: QwpTableBuffer[] = []; + private byName = new Map(); + private current?: QwpTableBuffer; + private rows = 0; + + reset(): SenderBuffer { + this.tables = []; + this.byName = new Map(); + this.current = undefined; + this.rows = 0; + return this; + } + + table(table: string): SenderBuffer { + let t = this.byName.get(table); + if (!t) { + t = new QwpTableBuffer(table); + this.byName.set(table, t); + this.tables.push(t); + } + this.current = t; + return this; + } + + private require(): QwpTableBuffer { + if (!this.current) throw new Error("table name must be set before adding columns"); + return this.current; + } + + symbol(name: string, value: unknown): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_SYMBOL); + if (col) col.values.push(String(value)); + return this; + } + + intColumn(name: string, value: number): SenderBuffer { + if (!Number.isInteger(value)) throw new Error(`value must be an integer, received ${value}`); + const col = this.require().getOrCreateColumn(name, TYPE_LONG); + if (col) col.values.push(BigInt(value)); + return this; + } + + floatColumn(name: string, value: number): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_DOUBLE); + if (col) col.values.push(value); + return this; + } + + timestampColumn(name: string, value: number | bigint, unit: TimestampUnit = "us"): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_TIMESTAMP); + if (col) col.values.push(toMicros(value, unit)); + return this; + } + + at(timestamp: number | bigint, unit: TimestampUnit = "us"): void { + const t = this.require(); + const col = t.getOrCreateColumn("timestamp", TYPE_TIMESTAMP); + if (col) col.values.push(toMicros(timestamp, unit)); + t.nextRow(); + this.rows++; + this.current = undefined; + } + + atNow(): void { + const t = this.require(); + t.nextRow(); + this.rows++; + this.current = undefined; + } + + toBufferNew(): Buffer | null { + const dirty = this.tables.filter((t) => t.rowCount > 0); + if (dirty.length === 0) return null; + const frame = encodeFrame(dirty); + this.reset(); + return frame; + } + + toBufferView(): Buffer { + throw new Error("toBufferView is not supported by the QWP buffer"); + } + + currentPosition(): number { + return this.rows; + } + + // Column types arriving in a later plan — fail loudly rather than emit wrong bytes. + stringColumn(): SenderBuffer { + return unsupported("stringColumn"); + } + booleanColumn(): SenderBuffer { + return unsupported("booleanColumn"); + } + arrayColumn(): SenderBuffer { + return unsupported("arrayColumn"); + } + decimalColumnText(): SenderBuffer { + return unsupported("decimalColumnText"); + } + decimalColumn(): SenderBuffer { + return unsupported("decimalColumn"); + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/buffer.test.ts && npx tsc --noEmit` +Expected: PASS, 5 tests, and no type errors — `tsc` proves `QwpBuffer` satisfies `SenderBuffer`. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/buffer.ts test/qwp/buffer.test.ts +git commit -m "feat(qwp): add QwpBuffer implementing SenderBuffer" +``` + +--- + +### Task 10: `QwpTransport` implementing `SenderTransport` + +**Files:** +- Create: `src/qwp/transport.ts` +- Test: `test/qwp/transport.test.ts` + +**Interfaces:** +- Consumes: `ws/socket.ts`. +- Produces: `class QwpTransport implements SenderTransport`. + +`getDefaultAutoFlushRows()` returns **1000** per spec 9.1 — not the ILP defaults. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/transport.test.ts +import { describe, it, expect } from "vitest"; +import { QwpTransport } from "../../src/qwp/transport"; +import { SenderOptions } from "../../src/options"; + +describe("QwpTransport", () => { + it("uses the QWP auto-flush row default, not the ILP one", () => { + const t = new QwpTransport(new SenderOptions("ws::addr=localhost:9000;")); + expect(t.getDefaultAutoFlushRows()).toBe(1000); + }); + + it("refuses to send before connect", async () => { + const t = new QwpTransport(new SenderOptions("ws::addr=localhost:9000;")); + await expect(t.send(Buffer.from([1]))).rejects.toThrow(/not connected/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/transport.test.ts` +Expected: FAIL — module not found (and `ws::` not yet accepted by `SenderOptions`; Task 11 fixes that, so this test is expected to stay red until then. If it fails on the protocol rather than the module, proceed to Task 11 and re-run.) + +- [ ] **Step 3: Write minimal implementation** + +```ts +// src/qwp/transport.ts +import { Buffer } from "node:buffer"; +import { SenderTransport } from "../transport"; +import { SenderOptions } from "../options"; +import { QwpWebSocket } from "./ws/socket"; + +const QWP_DEFAULT_AUTO_FLUSH_ROWS = 1000; // spec 9.1 +const CLIENT_ID = "nodejs/1.0.0"; // protocol client version, not the package version (spec 6.5) + +export class QwpTransport implements SenderTransport { + private readonly options: SenderOptions; + private ws?: QwpWebSocket; + + constructor(options: SenderOptions) { + this.options = options; + } + + async connect(): Promise { + const auth = this.options.username && this.options.password + ? "Basic " + + Buffer.from(`${this.options.username}:${this.options.password}`).toString("base64") + : this.options.token + ? `Bearer ${this.options.token}` + : undefined; + + this.ws = await QwpWebSocket.connect({ + host: this.options.host!, + port: this.options.port!, + tls: this.options.protocol === "wss", + clientId: CLIENT_ID, + authorization: auth, + rejectUnauthorized: this.options.tls_verify !== "unsafe_off", + }); + return true; + } + + async send(data: Buffer): Promise { + if (!this.ws) throw new Error("QWP transport is not connected"); + await this.ws.sendBinary(data); + return true; + } + + async close(): Promise { + await this.ws?.close(); + this.ws = undefined; + } + + getDefaultAutoFlushRows(): number { + return QWP_DEFAULT_AUTO_FLUSH_ROWS; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/transport.test.ts` +Expected: still failing on `ws::` protocol parsing until Task 11. That is expected; Task 11 ends with this test green. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/transport.ts test/qwp/transport.test.ts +git commit -m "feat(qwp): add QwpTransport implementing SenderTransport" +``` + +--- + +### Task 11: Wire `ws://` into options, buffer and transport factories + +**Files:** +- Modify: `src/options.ts` (4 sites), `src/buffer/index.ts`, `src/transport/index.ts`, `src/index.ts` +- Test: `test/qwp/options.test.ts` + +**Interfaces:** +- Consumes: `QwpBuffer`, `QwpTransport`. +- Produces: `WS = "ws"`, `WSS = "wss"` exported from `src/options.ts`. + +Implements spec 3.5. **The ordering hazard is the point of this task:** `createBuffer` must branch on `options.protocol` *before* its `protocol_version` switch, or a `ws::` sender silently gets `SenderBufferV1` and emits ILP text. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/options.test.ts +import { describe, it, expect } from "vitest"; +import { SenderOptions } from "../../src/options"; +import { createBuffer } from "../../src/buffer"; +import { createTransport } from "../../src/transport"; +import { QwpBuffer } from "../../src/qwp/buffer"; +import { QwpTransport } from "../../src/qwp/transport"; + +describe("ws:// wiring", () => { + it("accepts ws:: and defaults the port to 9000", () => { + const o = new SenderOptions("ws::addr=localhost;"); + expect(o.protocol).toBe("ws"); + expect(o.port).toBe(9000); + }); + + it("accepts wss:: ", () => { + expect(new SenderOptions("wss::addr=localhost;").protocol).toBe("wss"); + }); + + it("gives a ws:: sender a QwpBuffer, never an ILP buffer", () => { + const o = new SenderOptions("ws::addr=localhost:9000;"); + expect(createBuffer(o)).toBeInstanceOf(QwpBuffer); + }); + + it("gives a ws:: sender a QwpTransport", () => { + const o = new SenderOptions("ws::addr=localhost:9000;"); + expect(createTransport(o)).toBeInstanceOf(QwpTransport); + }); + + it("rejects protocol_version for ws:: (spec 9.2)", () => { + expect(() => new SenderOptions("ws::addr=localhost:9000;protocol_version=2;")).toThrow( + /not supported for WebSocket/i, + ); + }); + + it("still rejects a genuinely unknown protocol", () => { + expect(() => new SenderOptions("wsx::addr=localhost;")).toThrow(/invalid protocol/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/options.test.ts` +Expected: FAIL — `Invalid protocol: 'ws'`. + +- [ ] **Step 3: Apply the four edits** + +In `src/options.ts`: + +```ts +// (a) near the existing protocol constants +const WS = "ws"; +const WSS = "wss"; +``` + +```ts +// (b) in the protocol token switch inside the constructor + switch (options.protocol) { + case HTTP: + case HTTPS: + case TCP: + case TCPS: + case WS: + case WSS: + break; + default: + throw new Error( + `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps', 'ws', 'wss'`, + ); + } +``` + +```ts +// (c) parseProtocolVersion — leave ws/wss unset, and reject an explicit value +function parseProtocolVersion(options: SenderOptions) { + if (options.protocol === WS || options.protocol === WSS) { + if (options.protocol_version !== undefined && options.protocol_version !== null) { + throw new Error("protocol version is not supported for WebSocket protocol"); + } + return; // stays undefined: createBuffer branches on protocol first + } + // ...existing body unchanged +} +``` + +```ts +// (d) parseAddress port defaulting + switch (options.protocol) { + case HTTP: + case HTTPS: + case WS: + case WSS: + options.port = HTTP_PORT; // QWP shares the HTTP port, 9000 + return; + case TCP: + case TCPS: + options.port = TCP_PORT; + return; + default: + throw new Error( + `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps', 'ws', 'wss'`, + ); + } +``` + +Export them: add `WS, WSS` to the existing export list at the bottom of `src/options.ts`. + +In `src/buffer/index.ts` — **protocol check first**: + +```ts +import { SenderOptions, WS, WSS, /* ...existing */ } from "../options"; +import { QwpBuffer } from "../qwp/buffer"; + +function createBuffer(options: SenderOptions): SenderBuffer { + // QWP has no protocol_version; this MUST precede the version switch or a + // ws:// sender silently receives SenderBufferV1 and emits ILP text. + if (options.protocol === WS || options.protocol === WSS) { + return new QwpBuffer(); + } + switch (options.protocol_version) { + // ...existing arms unchanged + } +} +``` + +In `src/transport/index.ts`: + +```ts +import { SenderOptions, HTTP, HTTPS, TCP, TCPS, WS, WSS } from "../options"; +import { QwpTransport } from "../qwp/transport"; + + switch (options.protocol) { + case HTTP: + case HTTPS: + return options.stdlib_http ? new HttpTransport(options) : new UndiciTransport(options); + case TCP: + case TCPS: + return new TcpTransport(options); + case WS: + case WSS: + return new QwpTransport(options); + default: + throw new Error(`Invalid protocol: '${options.protocol}'`); + } +``` + +In `src/index.ts`: + +```ts +export { QwpBuffer } from "./qwp/buffer"; +export { QwpTransport } from "./qwp/transport"; +``` + +- [ ] **Step 4: Run the full suite** + +Run: `npx vitest run test/qwp/ && npx vitest run && npx tsc --noEmit && npx eslint src/**` +Expected: all `test/qwp/` green including Task 10's transport test, **all pre-existing tests still green**, no type errors, no lint errors. If `sender.config.test.ts` fails on an "accepted protocols" message, update that assertion to the new string — the spec predicted this at 3.5. + +- [ ] **Step 5: Commit** + +```bash +git add src/options.ts src/buffer/index.ts src/transport/index.ts src/index.ts test/qwp/options.test.ts +git commit -m "feat(qwp): route ws:// and wss:// to the QWP buffer and transport" +``` + +--- + +### Task 12: End-to-end against a real QuestDB + +**Files:** +- Create: `test/qwp/integration.test.ts` + +**Interfaces:** +- Consumes: everything above, via the public `Sender` API. + +This is the gate the whole plan exists to reach: a real server accepts our bytes. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/integration.test.ts +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { GenericContainer, StartedTestContainer } from "testcontainers"; +import { Sender } from "../../src"; + +let container: StartedTestContainer; +let httpPort: number; + +async function query(sql: string): Promise { + const res = await fetch( + `http://${container.getHost()}:${httpPort}/exec?query=${encodeURIComponent(sql)}`, + ); + return res.json(); +} + +describe("QWP ingest end-to-end", () => { + beforeAll(async () => { + // Matches the existing test/sender.integration.test.ts pattern: no wait + // strategy, readiness is established by the polling loop below. + container = await new GenericContainer("questdb/questdb:nightly") + .withExposedPorts(9000) + .start(); + httpPort = container.getMappedPort(9000); + }, 180_000); + + afterAll(async () => await container?.stop()); + + it("ingests rows over ws:// and they land with correct values", async () => { + // fromConfig is async. Do NOT pass auto_flush=off: spec 9.2 records that + // disabling auto-flush is rejected for WebSocket. The default triggers are + // harmless here because we flush explicitly and then poll. + const sender = await Sender.fromConfig( + `ws::addr=${container.getHost()}:${httpPort};`, + ); + await sender.connect(); + + await sender + .table("qwp_e2e") + .symbol("sym", "ETH-USD") + .floatColumn("price", 2615.54) + .intColumn("qty", 7) + .at(1_700_000_000_000_000n, "us"); + + await sender.flush(); + await sender.close(); + + // WAL apply is asynchronous — poll rather than sleeping a fixed interval. + let rows: any[] = []; + for (let i = 0; i < 60; i++) { + const r = await query("select sym, price, qty from qwp_e2e"); + rows = r.dataset ?? []; + if (rows.length > 0) break; + await new Promise((r) => setTimeout(r, 500)); + } + + expect(rows.length).toBe(1); + expect(rows[0][0]).toBe("ETH-USD"); + expect(rows[0][1]).toBeCloseTo(2615.54, 5); + expect(rows[0][2]).toBe(7); + }, 180_000); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/integration.test.ts` +Expected: FAIL. Confirm *why* before proceeding — a `Sender.connect()` error means wiring, a server NACK or dropped connection means the frame bytes are wrong. If the server closes without a response, dump the frame with `console.log(frame.toString("hex"))` in `QwpTransport.send` and compare against spec 6.2 by hand. + +- [ ] **Step 3: Fix whatever the failure reveals** + +No new code is planned here — Tasks 1–11 should already be sufficient. Expected failure modes and where to look: + +| Symptom | Likely cause | +|---|---| +| Upgrade fails with 404 | `WRITE_PATH` wrong, or QWP ingress disabled in the image | +| Server closes immediately after the frame | header magic/version/flags wrong (Task 4) | +| Rows land with shifted values | null-header or value-compaction bug (Task 4, spec 6.2.1) | +| Symbol column empty | inline dictionary index encoding (Task 4) | + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/integration.test.ts` +Expected: PASS, 1 test. + +- [ ] **Step 5: Run everything and commit** + +```bash +npx vitest run && npx tsc --noEmit && npx eslint src/** +git add test/qwp/integration.test.ts +git commit -m "test(qwp): add end-to-end ws:// ingest test against QuestDB" +``` + +--- + +## Self-Review + +**1. Spec coverage for PRs 1–3.** Frame codec + defrag + control frames + masking (Tasks 5, 6, 8 — spec 3.2.1, 3.2.2). Handshake + upgrade classification (Task 7 — spec 6.5, 6.5.1). Header + varint + four types + null bitmap + compaction (Tasks 1, 2, 4 — spec 6.0–6.3, 6.2.1). Row lifecycle rules (Task 3 — spec 6.5.3). Options wiring, all four sites + the `createBuffer` ordering hazard + `protocol_version` rejection (Task 11 — spec 3.5, 9.2). QWP auto-flush row default (Task 10 — spec 9.1). e2e (Task 12). + +**Deliberately deferred, with the spec section that covers them:** TLS trust-store mapping (6.5.2 — `tls_verify` is wired in Task 10, `tls_roots`/PEM/PKCS#12 is not), cap-splitting (5.1), commit frame (5.1.1), `auto_flush_bytes` and the per-transport interval hook (9.1), the three callback surfaces (4.2), connect-mode derivation (4.3), `reset()` semantics (4.1), row rollback on a throwing setter (4.1.1). Each belongs to Plan 2 or later; none is silently dropped. + +**2. Placeholder scan.** No TBDs. Every code step carries complete code. Task 12 Step 3 is the one step without pre-written code — deliberately, since it is a debugging step whose content depends on the failure, and it ships a symptom→cause table rather than "handle errors". + +**3. Type consistency.** `writeVarint`/`varintSize`/`readVarint` (Task 1) are used with those exact names in Task 4. `ColumnBuffer.values/nulls/size` (Task 3) are consumed with those names in Task 4. `encodeFrame(tables)` (Task 4) is called in Task 9. `QwpWebSocket.connect/sendBinary/close/maxBatchSize` (Task 8) are used in Task 10. `WS`/`WSS` (Task 11) are imported in the same task's edits to `buffer/index.ts` and `transport/index.ts`. `QwpBuffer` satisfies `SenderBuffer` — enforced by `tsc --noEmit` in Task 9 Step 4, not by inspection. + +**Known ordering wrinkle, stated rather than hidden:** Task 10's test cannot pass until Task 11 lands, because `SenderOptions` rejects `ws::` until then. Task 10 Step 4 says so explicitly and Task 11 Step 4 re-runs it. Reordering would require `QwpTransport` to exist before the factory that returns it, which is worse. + +**4. API assumptions verified against the current `main`** — three defects were found and fixed during this review rather than left for the implementer: + +- `Sender.fromConfig` is **`static async`**; the integration test now `await`s it. +- The integration test originally passed `auto_flush=off`, which spec 9.2 records as **rejected for WebSocket**. Removed. +- It also used `Wait.forLogMessage` with a guessed log pattern. The existing `test/sender.integration.test.ts` uses no wait strategy, so the plan now follows that pattern and relies on its own polling loop. + +Confirmed sound: `Sender.connect(): Promise` exists and is the right call for a connection-oriented transport; `new SenderOptions(configString)` reaches `parseProtocolVersion` via `parseConfigurationString`, so Task 11's constructor-throws test is valid; and `testcontainers` is already a devDependency. + +**Still unverified, and flagged as the plan's main risk:** that `questdb/questdb:nightly` has QWP ingress enabled on port 9000 by default. Task 12 Step 2 says to confirm the failure reason before changing code, and a 404 on the upgrade points here rather than at our bytes. If the image needs a flag, that is a container-config fix in Task 12, not a redesign. From 34b40d108e8790999b8bc9246ce010e56703fcbe Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 22:13:10 +0100 Subject: [PATCH 035/121] docs: add implementation plans 2 to 4 and a plan index Completes the four-plan stack covering spec PRs 1 to 16, decomposed so each plan produces working software rather than one unusable document. Plan 2 (PRs 4-8) completes the codec: remaining column types, VARCHAR and BINARY, arrays, decimals and geohash with their accumulation locks, row rollback, the symbol dictionary in both modes, Gorilla, and cap-splitting with the commit frame. Opens with the multi-frame contract widening that spec 3.1 predicted, done as its own reviewable change with behaviour unchanged. Plan 3 (PRs 9-11) adds response decoding, error categories and policies, the ACK-to-FSN correlation, the poison detector, the addr grammar, the host tracker and reconnect with dictionary catch-up. It carries a sequencing correction: the spec lists replay under PR 9, but replay needs the retention ring that Plan 4 builds, so this plan surfaces in-flight loss explicitly instead of hiding it, and Plan 4 removes that emission when replay lands. Plan 4 (PRs 12-16) adds CRC32C, the SF01 segment with torn-tail recovery, the FSN-keyed ring, replay, the alternating-generation boundary records, the persisted symbol dictionary, the delta-to-full-dict fallback, slot locking and crash tests, then ships 4.3.0. Its self-review names three items reduced in scope rather than implying parity: orphan drainers, slot quarantine, and the periodic fsync cadence. Adds a plans README listing the ten traps where the obvious implementation is the inverse of the correct one, so whoever executes these meets them before writing code rather than after. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-08-07-qwp-plan-2-full-codec.md | 1626 +++++++++++++++++ ...26-08-07-qwp-plan-3-errors-and-failover.md | 1364 ++++++++++++++ ...2026-08-07-qwp-plan-4-store-and-forward.md | 1196 ++++++++++++ docs/superpowers/plans/README.md | 55 + 4 files changed, 4241 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-07-qwp-plan-2-full-codec.md create mode 100644 docs/superpowers/plans/2026-08-07-qwp-plan-3-errors-and-failover.md create mode 100644 docs/superpowers/plans/2026-08-07-qwp-plan-4-store-and-forward.md create mode 100644 docs/superpowers/plans/README.md diff --git a/docs/superpowers/plans/2026-08-07-qwp-plan-2-full-codec.md b/docs/superpowers/plans/2026-08-07-qwp-plan-2-full-codec.md new file mode 100644 index 0000000..0c7601e --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-qwp-plan-2-full-codec.md @@ -0,0 +1,1626 @@ +# QWP Plan 2 — Full Codec (spec PRs 4–8) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Complete the QWP wire codec — every column type, the symbol dictionary in both modes, Gorilla timestamps, deferred commit with its commit frame, and cap-splitting when a flush exceeds the server's batch size. + +**Architecture:** Extends `src/qwp/` from Plan 1. Adds a per-type encoder table, a connection-scoped symbol dictionary, a Gorilla bit-writer, and the first genuine widening of the internal contract: a flush can now produce **several** frames, so `QwpBuffer` gains `sealFrames()` and `QwpTransport` gains `sendFrames()`. + +**Tech Stack:** TypeScript, Node ≥ 20, `node:buffer`, `node:crypto`. vitest + testcontainers. + +**Prerequisite:** Plan 1 (`docs/superpowers/plans/2026-08-07-qwp-plan-1-walking-skeleton.md`) must be merged. This plan consumes from it: `writeVarint`/`varintSize`/`readVarint`, `QwpTableBuffer`/`ColumnBuffer`, `encodeFrame`, `QwpBuffer`, `QwpTransport`, `QwpWebSocket`, and the constants module. + +**Source of truth:** `docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md`. + +## Global Constraints + +- **No new runtime dependencies.** +- **Node 20 floor.** No zstd on this path (spec 9.3) — do not add compression. +- **All integers little-endian.** `varint` is unsigned LEB128 (spec 6.0). +- **`V` = non-null row count.** Every column payload is compacted (spec 6.2.1). +- **Options stay `undefined` until set** (spec 9.1.2). +- Existing tests must stay green: `npx vitest run && npx tsc --noEmit && npx eslint src/**`. +- Out of scope: ACK handling, FSN, reconnect, failover, store-and-forward. Frames are still fire-and-forget after `sendFrames()`. + +## File Structure + +| File | Responsibility | +|---|---| +| `src/qwp/protocol/constants.ts` | **modify** — remaining type codes, encoding-byte constants | +| `src/qwp/protocol/columnWriter.ts` | **new** — per-type sizing and writing, extracted from `frameEncoder` | +| `src/qwp/protocol/bits.ts` | **new** — LSB-first bit writer | +| `src/qwp/protocol/gorilla.ts` | **new** — delta-of-delta encoder + feasibility check | +| `src/qwp/protocol/symbolDict.ts` | **new** — connection-scoped global dictionary | +| `src/qwp/protocol/frameEncoder.ts` | **modify** — flags, delta-dict section, commit frame | +| `src/qwp/buffer.ts` | **modify** — all column types, `sealFrames()`, row rollback | +| `src/qwp/transport.ts` | **modify** — `sendFrames()`, cap tracking | +| `src/sender.ts` | **modify** — multi-frame flush path | + +--- + +### Task 1: Widen the flush contract to multiple frames + +**Files:** +- Modify: `src/qwp/buffer.ts`, `src/qwp/transport.ts`, `src/sender.ts` +- Test: `test/qwp/multiframe.test.ts` + +**Interfaces:** +- Consumes: Plan 1's `QwpBuffer`, `QwpTransport`. +- Produces: `interface QwpMultiFrame { sealFrames(maxBatchSize: number): Buffer[] }` on `QwpBuffer`; `sendFrames(frames: Buffer[]): Promise` on `QwpTransport`; a branch in `Sender.flush()`. + +This is the widening spec 3.1 predicted. Do it before cap-splitting needs it, as its own reviewable change with behaviour unchanged (one frame in, one frame out). + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/multiframe.test.ts +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../src/qwp/buffer"; + +describe("multi-frame sealing", () => { + it("returns a single frame when the batch fits", () => { + const b = new QwpBuffer(); + b.table("t").intColumn("x", 1); + b.at(1n, "us"); + const frames = b.sealFrames(1_000_000); + expect(frames.length).toBe(1); + expect(frames[0].subarray(0, 4).toString("ascii")).toBe("QWP1"); + }); + + it("returns an empty array when nothing is buffered", () => { + expect(new QwpBuffer().sealFrames(1_000_000)).toEqual([]); + }); + + it("clears state after sealing", () => { + const b = new QwpBuffer(); + b.table("t").intColumn("x", 1); + b.at(1n, "us"); + b.sealFrames(1_000_000); + expect(b.sealFrames(1_000_000)).toEqual([]); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/multiframe.test.ts` +Expected: FAIL — `b.sealFrames is not a function`. + +- [ ] **Step 3: Implement** + +In `src/qwp/buffer.ts`, replace `toBufferNew` and add `sealFrames`: + +```ts + /** + * Seals the buffered rows into one or more frames. A flush produces more + * than one frame only when the encoded batch exceeds maxBatchSize (spec 5.1); + * splitting itself lands in Task 9. + */ + sealFrames(maxBatchSize: number): Buffer[] { + const dirty = this.tables.filter((t) => t.rowCount > 0); + if (dirty.length === 0) return []; + const frame = encodeFrame(dirty); + this.reset(); + return [frame]; + } + + toBufferNew(): Buffer | null { + const frames = this.sealFrames(Number.MAX_SAFE_INTEGER); + if (frames.length === 0) return null; + if (frames.length > 1) { + throw new Error("QWP produced multiple frames; use sealFrames()"); + } + return frames[0]; + } +``` + +In `src/qwp/transport.ts`: + +```ts + /** Sends each frame as its own WebSocket binary message. */ + async sendFrames(frames: Buffer[]): Promise { + if (!this.ws) throw new Error("QWP transport is not connected"); + for (const f of frames) { + await this.ws.sendBinary(f); + } + return true; + } + + /** Server-advertised cap, or a conservative default before the handshake. */ + get maxBatchSize(): number { + return this.ws?.maxBatchSize ?? 16 * 1024 * 1024; + } +``` + +In `src/sender.ts`, replace the body of `flush()`: + +```ts + async flush(): Promise { + // QWP can produce several frames per flush; ILP always produces one. + const buf = this.buffer as unknown as { + sealFrames?: (cap: number) => Buffer[]; + }; + const tx = this.transport as unknown as { + sendFrames?: (f: Buffer[]) => Promise; + maxBatchSize?: number; + }; + if (buf.sealFrames && tx.sendFrames) { + const frames = buf.sealFrames(tx.maxBatchSize ?? Number.MAX_SAFE_INTEGER); + if (frames.length === 0) return false; + this.log("debug", `Flushing ${frames.length} QWP frame(s)`); + this.resetAutoFlush(); + await tx.sendFrames(frames); + return true; + } + + const dataToSend: Buffer = this.buffer.toBufferNew(); + if (!dataToSend) { + return false; // Nothing to send + } + this.log( + "debug", + `Flushing, number of flushed rows: ${this.pendingRowCount}`, + ); + this.resetAutoFlush(); + await this.transport.send(dataToSend); + return true; + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx vitest run && npx tsc --noEmit` +Expected: PASS. ILP tests unaffected — they take the original branch. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/buffer.ts src/qwp/transport.ts src/sender.ts test/qwp/multiframe.test.ts +git commit -m "refactor(qwp): allow a flush to produce multiple frames" +``` + +--- + +### Task 2: Remaining fixed-width column types + +**Files:** +- Modify: `src/qwp/protocol/constants.ts` +- Create: `src/qwp/protocol/columnWriter.ts` +- Modify: `src/qwp/protocol/frameEncoder.ts` +- Test: `test/qwp/columnWriter.test.ts` + +**Interfaces:** +- Produces: `columnPayloadSize(col, rowCount, opts): number`, `writeColumn(buf, offset, col, rowCount, opts): number` exported from `columnWriter.ts`; `frameEncoder` delegates to them. `interface EncodeOpts { gorilla: boolean; globalSymbols?: SymbolDict }`. + +Adds BOOLEAN (bit-packed), BYTE, SHORT, INT, FLOAT, DATE, UUID, LONG256, CHAR, IPv4 per spec 6.3. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/columnWriter.test.ts +import { describe, it, expect } from "vitest"; +import { columnPayloadSize, writeColumn } from "../../src/qwp/protocol/columnWriter"; +import { TYPE_BOOLEAN, TYPE_INT, TYPE_FLOAT } from "../../src/qwp/protocol/constants"; + +const opts = { gorilla: false }; + +function encode(col: any, rowCount: number): Buffer { + const size = columnPayloadSize(col, rowCount, opts); + const b = Buffer.alloc(size); + const end = writeColumn(b, 0, col, rowCount, opts); + expect(end).toBe(size); + return b; +} + +describe("column writers", () => { + it("bit-packs BOOLEAN LSB-first over non-null values", () => { + const col = { name: "b", type: TYPE_BOOLEAN, values: [true, false, true], nulls: [false, false, false], size: 3 }; + const b = encode(col, 3); + expect(b[0]).toBe(0); // nullHeader + expect(b[1]).toBe(0b00000101); // bits 0 and 2 + expect(b.length).toBe(2); + }); + + it("writes INT as 4 bytes LE", () => { + const col = { name: "i", type: TYPE_INT, values: [258], nulls: [false], size: 1 }; + const b = encode(col, 1); + expect(b.readInt32LE(1)).toBe(258); + }); + + it("writes FLOAT as 4 bytes IEEE754", () => { + const col = { name: "f", type: TYPE_FLOAT, values: [1.5], nulls: [false], size: 1 }; + const b = encode(col, 1); + expect(b.readFloatLE(1)).toBeCloseTo(1.5, 6); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/columnWriter.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +Add to `src/qwp/protocol/constants.ts`: + +```ts +export const TYPE_BOOLEAN = 0x01; +export const TYPE_BYTE = 0x02; +export const TYPE_SHORT = 0x03; +export const TYPE_INT = 0x04; +export const TYPE_FLOAT = 0x06; +export const TYPE_DATE = 0x0b; +export const TYPE_UUID = 0x0c; +export const TYPE_LONG256 = 0x0d; +export const TYPE_GEOHASH = 0x0e; +export const TYPE_VARCHAR = 0x0f; +export const TYPE_TIMESTAMP_NANOS = 0x10; +export const TYPE_DOUBLE_ARRAY = 0x11; +export const TYPE_LONG_ARRAY = 0x12; +export const TYPE_DECIMAL64 = 0x13; +export const TYPE_DECIMAL128 = 0x14; +export const TYPE_DECIMAL256 = 0x15; +export const TYPE_CHAR = 0x16; +export const TYPE_BINARY = 0x17; +export const TYPE_IPV4 = 0x18; + +export const ENCODING_UNCOMPRESSED = 0x00; +export const ENCODING_GORILLA = 0x01; +``` + +Create `src/qwp/protocol/columnWriter.ts`. Move the null-header and bitmap logic out of `frameEncoder.ts` verbatim, then add the new arms: + +```ts +// src/qwp/protocol/columnWriter.ts +import { Buffer } from "node:buffer"; +import { writeVarint, varintSize } from "./varint"; +import { ColumnBuffer } from "./tableBuffer"; +import * as T from "./constants"; + +export interface EncodeOpts { + gorilla: boolean; +} + +function nullCountOf(col: ColumnBuffer): number { + let n = 0; + for (const v of col.nulls) if (v) n++; + return n; +} + +function fixedWidth(type: number): number | undefined { + switch (type) { + case T.TYPE_BYTE: + return 1; + case T.TYPE_SHORT: + case T.TYPE_CHAR: + return 2; + case T.TYPE_INT: + case T.TYPE_FLOAT: + case T.TYPE_IPV4: + return 4; + case T.TYPE_LONG: + case T.TYPE_DOUBLE: + case T.TYPE_DATE: + return 8; + case T.TYPE_UUID: + return 16; + case T.TYPE_LONG256: + return 32; + default: + return undefined; + } +} + +export function columnPayloadSize( + col: ColumnBuffer, + rowCount: number, + opts: EncodeOpts, +): number { + let n = 1; + if (nullCountOf(col) > 0) n += Math.ceil(rowCount / 8); + const v = col.values.length; + + if (col.type === T.TYPE_BOOLEAN) return n + Math.ceil(v / 8); + + const w = fixedWidth(col.type); + if (w !== undefined) return n + v * w; + + if (col.type === T.TYPE_SYMBOL) { + const dict = [...new Set(col.values as string[])]; + n += varintSize(dict.length); + for (const s of dict) { + const b = Buffer.byteLength(s, "utf8"); + n += varintSize(b) + b; + } + for (const s of col.values as string[]) n += varintSize(dict.indexOf(s)); + return n; + } + + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); +} + +export function writeColumn( + buf: Buffer, + offset: number, + col: ColumnBuffer, + rowCount: number, + opts: EncodeOpts, +): number { + let o = offset; + if (nullCountOf(col) > 0) { + buf[o++] = 1; + const bytes = Math.ceil(rowCount / 8); + buf.fill(0, o, o + bytes); + for (let i = 0; i < rowCount; i++) { + if (col.nulls[i]) buf[o + (i >>> 3)] |= 1 << (i & 7); + } + o += bytes; + } else { + buf[o++] = 0; + } + + switch (col.type) { + case T.TYPE_BOOLEAN: { + const bytes = Math.ceil(col.values.length / 8); + buf.fill(0, o, o + bytes); + col.values.forEach((v, i) => { + if (v) buf[o + (i >>> 3)] |= 1 << (i & 7); + }); + return o + bytes; + } + case T.TYPE_BYTE: + for (const v of col.values) buf.writeInt8(Number(v), o++); + return o; + case T.TYPE_SHORT: + for (const v of col.values) { + buf.writeInt16LE(Number(v), o); + o += 2; + } + return o; + case T.TYPE_CHAR: + for (const v of col.values) { + buf.writeUInt16LE((v as string).charCodeAt(0), o); + o += 2; + } + return o; + case T.TYPE_INT: + for (const v of col.values) { + buf.writeInt32LE(Number(v), o); + o += 4; + } + return o; + case T.TYPE_IPV4: + for (const v of col.values) { + buf.writeUInt32LE(Number(v) >>> 0, o); + o += 4; + } + return o; + case T.TYPE_FLOAT: + for (const v of col.values) { + buf.writeFloatLE(Number(v), o); + o += 4; + } + return o; + case T.TYPE_LONG: + case T.TYPE_DATE: + for (const v of col.values) { + buf.writeBigInt64LE(BigInt(v as number | bigint), o); + o += 8; + } + return o; + case T.TYPE_DOUBLE: + for (const v of col.values) { + buf.writeDoubleLE(Number(v), o); + o += 8; + } + return o; + case T.TYPE_UUID: + for (const v of col.values) { + (v as unknown as Buffer).copy(buf, o); + o += 16; + } + return o; + case T.TYPE_LONG256: + for (const v of col.values) { + (v as unknown as Buffer).copy(buf, o); + o += 32; + } + return o; + case T.TYPE_SYMBOL: { + const dict = [...new Set(col.values as string[])]; + o = writeVarint(buf, o, dict.length); + for (const s of dict) { + const n = Buffer.byteLength(s, "utf8"); + o = writeVarint(buf, o, n); + buf.write(s, o, "utf8"); + o += n; + } + for (const s of col.values as string[]) o = writeVarint(buf, o, dict.indexOf(s)); + return o; + } + default: + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); + } +} +``` + +Then in `frameEncoder.ts`, delete the local `columnPayloadSize`/`writeColumn` and import them from `./columnWriter`, passing `{ gorilla: false }`. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS, including Plan 1's `frameEncoder.test.ts` unchanged. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/constants.ts src/qwp/protocol/columnWriter.ts src/qwp/protocol/frameEncoder.ts test/qwp/columnWriter.test.ts +git commit -m "feat(qwp): add remaining fixed-width column types" +``` + +--- + +### Task 3: VARCHAR and BINARY + +**Files:** +- Modify: `src/qwp/protocol/columnWriter.ts`, `src/qwp/buffer.ts` +- Test: `test/qwp/columnWriter.varchar.test.ts` + +Wire layout is `(V+1) × u32` offsets then concatenated bytes (spec 6.3). BINARY shares it exactly. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/columnWriter.varchar.test.ts +import { describe, it, expect } from "vitest"; +import { columnPayloadSize, writeColumn } from "../../src/qwp/protocol/columnWriter"; +import { TYPE_VARCHAR } from "../../src/qwp/protocol/constants"; + +describe("VARCHAR", () => { + it("writes V+1 offsets then concatenated utf8", () => { + const col = { name: "s", type: TYPE_VARCHAR, values: ["ab", "cde"], nulls: [false, false], size: 2 }; + const opts = { gorilla: false }; + const size = columnPayloadSize(col as any, 2, opts); + const b = Buffer.alloc(size); + expect(writeColumn(b, 0, col as any, 2, opts)).toBe(size); + expect(b[0]).toBe(0); // nullHeader + expect(b.readUInt32LE(1)).toBe(0); + expect(b.readUInt32LE(5)).toBe(2); + expect(b.readUInt32LE(9)).toBe(5); + expect(b.subarray(13).toString("utf8")).toBe("abcde"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/columnWriter.varchar.test.ts` +Expected: FAIL — "unsupported QWP column type: 0xf". + +- [ ] **Step 3: Implement** + +Add to `columnPayloadSize`, before the `throw`: + +```ts + if (col.type === T.TYPE_VARCHAR || col.type === T.TYPE_BINARY) { + let data = 0; + for (const s of col.values) { + data += col.type === T.TYPE_VARCHAR + ? Buffer.byteLength(s as string, "utf8") + : (s as unknown as Buffer).length; + } + return n + (v + 1) * 4 + data; + } +``` + +Add to `writeColumn`, before `default:`: + +```ts + case T.TYPE_VARCHAR: + case T.TYPE_BINARY: { + const parts: Buffer[] = col.values.map((s) => + col.type === T.TYPE_VARCHAR + ? Buffer.from(s as string, "utf8") + : (s as unknown as Buffer), + ); + let acc = 0; + buf.writeUInt32LE(0, o); + o += 4; + for (const p of parts) { + acc += p.length; + buf.writeUInt32LE(acc, o); + o += 4; + } + for (const p of parts) { + p.copy(buf, o); + o += p.length; + } + return o; + } +``` + +In `src/qwp/buffer.ts`, replace the `stringColumn` stub: + +```ts + stringColumn(name: string, value: string): SenderBuffer { + if (typeof value !== "string") throw new Error("stringColumn accepts only string values"); + const col = this.require().getOrCreateColumn(name, TYPE_VARCHAR); + if (col) col.values.push(value); + return this; + } +``` + +(Import `TYPE_VARCHAR` from `./protocol/constants`.) + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/columnWriter.ts src/qwp/buffer.ts test/qwp/columnWriter.varchar.test.ts +git commit -m "feat(qwp): add VARCHAR and BINARY columns" +``` + +--- + +### Task 4: Arrays, decimals, geohash — with their accumulation rules + +**Files:** +- Modify: `src/qwp/protocol/tableBuffer.ts`, `src/qwp/protocol/columnWriter.ts`, `src/qwp/buffer.ts` +- Test: `test/qwp/columnWriter.complex.test.ts` + +**Spec 6.5.3 rules that must be implemented, not just the wire format:** +- GEOHASH precision is **locked on the column's first value**, 1–60. +- DECIMAL scale is **locked on first value**, and later values are **rescaled**, throwing only on precision loss. +- Arrays reject jagged shapes. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/columnWriter.complex.test.ts +import { describe, it, expect } from "vitest"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { TYPE_GEOHASH, TYPE_DOUBLE_ARRAY } from "../../src/qwp/protocol/constants"; +import { columnPayloadSize, writeColumn } from "../../src/qwp/protocol/columnWriter"; + +describe("complex column rules", () => { + it("locks geohash precision on the first value", () => { + const t = new QwpTableBuffer("t"); + const c = t.getOrCreateColumn("g", TYPE_GEOHASH)!; + t.setGeoHashPrecision(c, 20); + expect(() => t.setGeoHashPrecision(c, 25)).toThrow(/precision mismatch/i); + }); + + it("rejects an out-of-range geohash precision", () => { + const t = new QwpTableBuffer("t"); + const c = t.getOrCreateColumn("g", TYPE_GEOHASH)!; + expect(() => t.setGeoHashPrecision(c, 61)).toThrow(/1-60/); + }); + + it("writes a double array as per-value shape then values", () => { + const col = { + name: "m", type: TYPE_DOUBLE_ARRAY, + values: [{ dims: [2], data: [1.5, 2.5] }], + nulls: [false], size: 1, + }; + const opts = { gorilla: false }; + const size = columnPayloadSize(col as any, 1, opts); + const b = Buffer.alloc(size); + expect(writeColumn(b, 0, col as any, 1, opts)).toBe(size); + expect(b[1]).toBe(1); // nDims + expect(b.readUInt32LE(2)).toBe(2); // dim length + expect(b.readDoubleLE(6)).toBeCloseTo(1.5); + expect(b.readDoubleLE(14)).toBeCloseTo(2.5); + }); + + it("rejects a jagged array", () => { + const { flattenArray } = require("../../src/qwp/protocol/columnWriter"); + expect(() => flattenArray([[1, 2], [3]])).toThrow(/irregular array shape/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/columnWriter.complex.test.ts` +Expected: FAIL — `t.setGeoHashPrecision is not a function`. + +- [ ] **Step 3: Implement** + +In `tableBuffer.ts`, extend `ColumnBuffer` and add the locks: + +```ts +export interface ColumnBuffer { + name: string; + type: number; + values: unknown[]; + nulls: boolean[]; + size: number; + geohashPrecision?: number; + decimalScale?: number; +} +``` + +```ts + /** Precision is 1-60 and locked on the column's first value (spec 6.5.3). */ + setGeoHashPrecision(col: ColumnBuffer, precision: number): void { + if (precision < 1 || precision > 60) { + throw new Error(`invalid GeoHash precision: ${precision} (must be 1-60)`); + } + if (col.geohashPrecision === undefined) { + col.geohashPrecision = precision; + } else if (col.geohashPrecision !== precision) { + throw new Error( + `GeoHash precision mismatch: column has ${col.geohashPrecision} bits, got ${precision}`, + ); + } + } + + /** Scale locks on the first value; later values rescale (spec 6.5.3). */ + setDecimalScale(col: ColumnBuffer, scale: number): number { + if (col.decimalScale === undefined) col.decimalScale = scale; + return col.decimalScale; + } +``` + +In `columnWriter.ts` add the shape helper and the arms: + +```ts +export function flattenArray(a: unknown[]): { dims: number[]; data: number[] } { + const dims: number[] = []; + let level: unknown = a; + while (Array.isArray(level)) { + dims.push(level.length); + level = level[0]; + } + const data: number[] = []; + const walk = (node: unknown, depth: number): void => { + if (depth === dims.length) { + data.push(node as number); + return; + } + if (!Array.isArray(node) || node.length !== dims[depth]) { + throw new Error("irregular array shape"); + } + for (const child of node) walk(child, depth + 1); + }; + walk(a, 0); + return { dims, data }; +} +``` + +Size arm: + +```ts + if (col.type === T.TYPE_DOUBLE_ARRAY || col.type === T.TYPE_LONG_ARRAY) { + let total = 0; + for (const val of col.values) { + const a = val as { dims: number[]; data: number[] }; + total += 1 + a.dims.length * 4 + a.data.length * 8; + } + return n + total; + } + if (col.type === T.TYPE_GEOHASH) { + const p = col.geohashPrecision ?? 1; + return n + varintSize(p) + v * Math.ceil(p / 8); + } + if (col.type === T.TYPE_DECIMAL64) return n + 1 + v * 8; + if (col.type === T.TYPE_DECIMAL128) return n + 1 + v * 16; + if (col.type === T.TYPE_DECIMAL256) return n + 1 + v * 32; +``` + +Write arms: + +```ts + case T.TYPE_DOUBLE_ARRAY: + case T.TYPE_LONG_ARRAY: { + for (const val of col.values) { + const a = val as { dims: number[]; data: number[] }; + buf.writeUInt8(a.dims.length, o++); + for (const d of a.dims) { + buf.writeUInt32LE(d, o); + o += 4; + } + for (const x of a.data) { + if (col.type === T.TYPE_DOUBLE_ARRAY) buf.writeDoubleLE(x, o); + else buf.writeBigInt64LE(BigInt(x), o); + o += 8; + } + } + return o; + } + case T.TYPE_GEOHASH: { + const p = col.geohashPrecision ?? 1; + o = writeVarint(buf, o, p); + const width = Math.ceil(p / 8); + for (const val of col.values) { + let bits = BigInt(val as bigint); + for (let i = 0; i < width; i++) { + buf.writeUInt8(Number(bits & 0xffn), o++); + bits >>= 8n; + } + } + return o; + } + case T.TYPE_DECIMAL64: + case T.TYPE_DECIMAL128: + case T.TYPE_DECIMAL256: { + // scale is one byte at the START of the column payload, not in the + // schema -- QwpConstants' javadoc says "in schema" and is wrong (spec 6.3) + buf.writeUInt8(col.decimalScale ?? 0, o++); + const width = col.type === T.TYPE_DECIMAL64 ? 8 : col.type === T.TYPE_DECIMAL128 ? 16 : 32; + for (const val of col.values) { + let x = BigInt(val as bigint); + for (let i = 0; i < width; i++) { + buf.writeUInt8(Number(x & 0xffn), o++); + x >>= 8n; + } + } + return o; + } +``` + +In `buffer.ts` replace the `arrayColumn` stub: + +```ts + arrayColumn(name: string, value: unknown[]): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_DOUBLE_ARRAY); + if (col) col.values.push(flattenArray(value)); + return this; + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS, 4 tests in the new file. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/tableBuffer.ts src/qwp/protocol/columnWriter.ts src/qwp/buffer.ts test/qwp/columnWriter.complex.test.ts +git commit -m "feat(qwp): add array, decimal and geohash columns with their locks" +``` + +--- + +### Task 5: Row rollback on a throwing setter + +**Files:** +- Modify: `src/qwp/buffer.ts` +- Test: `test/qwp/rollback.test.ts` + +Spec 4.1.1. A setter that throws mid-row must roll **all** columns back to the last row boundary, or columns desynchronise and every later frame is malformed while still looking structurally valid. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/rollback.test.ts +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../src/qwp/buffer"; + +describe("row rollback", () => { + it("leaves all columns equal-length after a mid-row throw", () => { + const b = new QwpBuffer(); + b.table("t").intColumn("a", 1); + expect(() => b.intColumn("bad", 1.5)).toThrow(); // not an integer + b.intColumn("a2", 2); + b.at(1n, "us"); + const frame = b.sealFrames(1_000_000)[0]; + // One row; the frame must encode without a size mismatch, which the + // encoder asserts internally. + expect(frame.readUInt16LE(6)).toBe(1); + }); + + it("produces bytes identical to a row that was never started", () => { + const a = new QwpBuffer(); + a.table("t").intColumn("x", 1); + a.at(5n, "us"); + const clean = a.sealFrames(1_000_000)[0]; + + const c = new QwpBuffer(); + c.table("t"); + expect(() => c.intColumn("x", 0.5)).toThrow(); + c.table("t").intColumn("x", 1); + c.at(5n, "us"); + expect(c.sealFrames(1_000_000)[0].equals(clean)).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/rollback.test.ts` +Expected: FAIL on the byte-equality assertion — the failed setter leaves a partial column. + +- [ ] **Step 3: Implement** + +In `QwpTableBuffer` add: + +```ts + /** Truncates every column back to the last completed row (spec 4.1.1). */ + rollbackRow(): void { + for (const c of this.cols) { + while (c.size > this.rows) { + const wasNull = c.nulls.pop(); + c.size--; + if (wasNull === false) c.values.pop(); + } + } + // Drop columns created solely by the abandoned row. + for (let i = this.cols.length - 1; i >= 0; i--) { + if (this.cols[i].size === 0 && this.rows === 0) { + this.byName.delete(this.cols[i].name); + this.cols.splice(i, 1); + } + } + } +``` + +In `QwpBuffer`, wrap every column setter. Introduce one helper and route all setters through it: + +```ts + private guard(fn: () => R): R { + try { + return fn(); + } catch (e) { + this.current?.rollbackRow(); + throw e; + } + } + + intColumn(name: string, value: number): SenderBuffer { + return this.guard(() => { + if (!Number.isInteger(value)) { + throw new Error(`value must be an integer, received ${value}`); + } + const col = this.require().getOrCreateColumn(name, TYPE_LONG); + if (col) col.values.push(BigInt(value)); + return this; + }); + } +``` + +Apply the same `this.guard(() => { ... })` wrapper to `symbol`, `floatColumn`, `timestampColumn`, `stringColumn`, `booleanColumn`, `arrayColumn`, `decimalColumnText`, `decimalColumn`. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS, 2 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/tableBuffer.ts src/qwp/buffer.ts test/qwp/rollback.test.ts +git commit -m "feat(qwp): roll back the in-progress row when a setter throws" +``` + +--- + +### Task 6: Symbol dictionary — full-dict mode + +**Files:** +- Create: `src/qwp/protocol/symbolDict.ts` +- Test: `test/qwp/symbolDict.test.ts` + +**Interfaces:** +- Produces: `class SymbolDict` with `getOrAdd(s: string): number`, `size(): number`, `entriesFrom(startId: number): string[]`, `reset(): void`, `addRecovered(s: string): number`. + +`addRecovered` appends **without de-duplicating** (spec 8.1.6) — ids are positional, and collapsing two entries silently renumbers everything after. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/symbolDict.test.ts +import { describe, it, expect } from "vitest"; +import { SymbolDict } from "../../src/qwp/protocol/symbolDict"; +import { MAX_SYMBOL_DICTIONARY_SIZE } from "../../src/qwp/protocol/constants"; + +describe("SymbolDict", () => { + it("assigns dense ids from 0 and de-dupes on getOrAdd", () => { + const d = new SymbolDict(); + expect(d.getOrAdd("a")).toBe(0); + expect(d.getOrAdd("b")).toBe(1); + expect(d.getOrAdd("a")).toBe(0); + expect(d.size()).toBe(2); + }); + + it("returns entries above a baseline", () => { + const d = new SymbolDict(); + d.getOrAdd("a"); + d.getOrAdd("b"); + d.getOrAdd("c"); + expect(d.entriesFrom(1)).toEqual(["b", "c"]); + }); + + it("addRecovered never de-duplicates", () => { + const d = new SymbolDict(); + d.addRecovered("x"); + d.addRecovered("x"); + expect(d.size()).toBe(2); // positional ids must be preserved + }); + + it("enforces the dictionary cap at registration time", () => { + const d = new SymbolDict(); + // Cheap proxy: assert the guard exists rather than adding a million entries. + expect(MAX_SYMBOL_DICTIONARY_SIZE).toBe(1_000_000); + expect(() => d.checkCap(MAX_SYMBOL_DICTIONARY_SIZE)).toThrow(/dictionary/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/symbolDict.test.ts` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement** + +Add `export const MAX_SYMBOL_DICTIONARY_SIZE = 1_000_000;` to `constants.ts`, then: + +```ts +// src/qwp/protocol/symbolDict.ts +import { MAX_SYMBOL_DICTIONARY_SIZE } from "./constants"; + +/** Connection-scoped global symbol dictionary. Ids are dense from 0. */ +export class SymbolDict { + private readonly ids = new Map(); + private readonly list: string[] = []; + + size(): number { + return this.list.length; + } + + checkCap(next: number): void { + if (next >= MAX_SYMBOL_DICTIONARY_SIZE) { + throw new Error( + `symbol dictionary exceeds maximum size ${MAX_SYMBOL_DICTIONARY_SIZE}`, + ); + } + } + + getOrAdd(s: string): number { + const existing = this.ids.get(s); + if (existing !== undefined) return existing; + this.checkCap(this.list.length); + const id = this.list.length; + this.ids.set(s, id); + this.list.push(s); + return id; + } + + /** + * Appends at the next id WITHOUT de-duplicating. The persisted dictionary, + * the wire delta and the catch-up mirror all key on POSITION, so collapsing + * two entries would leave this shorter than the persisted count and silently + * misattribute every later symbol (spec 8.1.6). + */ + addRecovered(s: string): number { + const id = this.list.length; + this.list.push(s); + if (!this.ids.has(s)) this.ids.set(s, id); + else this.ids.set(s, id); // keep the highest id; both encode identically + return id; + } + + entriesFrom(startId: number): string[] { + return this.list.slice(Math.max(0, startId)); + } + + reset(): void { + this.ids.clear(); + this.list.length = 0; + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/symbolDict.test.ts` +Expected: PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/symbolDict.ts src/qwp/protocol/constants.ts test/qwp/symbolDict.test.ts +git commit -m "feat(qwp): add connection-scoped symbol dictionary" +``` + +--- + +### Task 7: Delta symbol dictionary on the wire + +**Files:** +- Modify: `src/qwp/protocol/frameEncoder.ts`, `src/qwp/protocol/columnWriter.ts`, `src/qwp/buffer.ts` +- Test: `test/qwp/deltaDict.test.ts` + +Spec 5.2 and 6.2. With `FLAG_DELTA_SYMBOL_DICT` set, the payload opens with `varint deltaStart, varint deltaCount, count × [varint len][utf8]`, and symbol columns carry **no per-column dictionary** — just a varint global id per value. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/deltaDict.test.ts +import { describe, it, expect } from "vitest"; +import { encodeFrame } from "../../src/qwp/protocol/frameEncoder"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { SymbolDict } from "../../src/qwp/protocol/symbolDict"; +import { TYPE_SYMBOL, FLAG_DELTA_SYMBOL_DICT } from "../../src/qwp/protocol/constants"; +import { readVarint } from "../../src/qwp/protocol/varint"; + +describe("delta symbol dictionary", () => { + it("sets the flag and emits only newly-seen symbols", () => { + const dict = new SymbolDict(); + dict.getOrAdd("already"); // id 0, already confirmed + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("s", TYPE_SYMBOL)!.values.push(dict.getOrAdd("fresh")); // id 1 + t.nextRow(); + + const f = encodeFrame([t], { gorilla: false, dict, confirmedMaxId: 0 }); + expect(f.readUInt8(5) & FLAG_DELTA_SYMBOL_DICT).toBe(FLAG_DELTA_SYMBOL_DICT); + + let o = 12; + const start = readVarint(f, o); + o = start.offset; + const count = readVarint(f, o); + expect(start.value).toBe(1); // confirmedMaxId + 1 + expect(count.value).toBe(1); // only "fresh" + }); + + it("emits an empty delta when nothing new was registered", () => { + const dict = new SymbolDict(); + dict.getOrAdd("a"); + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("s", TYPE_SYMBOL)!.values.push(0); + t.nextRow(); + const f = encodeFrame([t], { gorilla: false, dict, confirmedMaxId: 0 }); + let o = 12; + const start = readVarint(f, o); + const count = readVarint(f, start.offset); + expect(count.value).toBe(0); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/deltaDict.test.ts` +Expected: FAIL — `encodeFrame` takes one argument. + +- [ ] **Step 3: Implement** + +Change `encodeFrame`'s signature and body in `frameEncoder.ts`: + +```ts +export interface FrameOpts { + gorilla: boolean; + /** Present means delta mode; absent means full-dict/inline mode. */ + dict?: SymbolDict; + /** Highest symbol id the server has already confirmed. */ + confirmedMaxId?: number; + deferCommit?: boolean; +} + +export function encodeFrame(tables: QwpTableBuffer[], opts: FrameOpts): Buffer { + const delta = opts.dict !== undefined; + const deltaStart = delta ? (opts.confirmedMaxId ?? -1) + 1 : 0; + const entries = delta ? opts.dict!.entriesFrom(deltaStart) : []; + + let flags = 0; + if (opts.gorilla) flags |= FLAG_GORILLA; + if (delta) flags |= FLAG_DELTA_SYMBOL_DICT; + if (opts.deferCommit) flags |= FLAG_DEFER_COMMIT; + + let payloadLen = 0; + if (delta) { + payloadLen += varintSize(deltaStart) + varintSize(entries.length); + for (const s of entries) { + const n = Buffer.byteLength(s, "utf8"); + payloadLen += varintSize(n) + n; + } + } + const colOpts = { gorilla: opts.gorilla, delta }; + payloadLen += tables.reduce((a, t) => a + tableSize(t, colOpts), 0); + + const buf = Buffer.allocUnsafe(HEADER_SIZE + payloadLen); + QWP_MAGIC.copy(buf, 0); + buf.writeUInt8(QWP_VERSION, 4); + buf.writeUInt8(flags, 5); + buf.writeUInt16LE(tables.length, 6); + buf.writeUInt32LE(payloadLen, 8); + + let o = HEADER_SIZE; + if (delta) { + o = writeVarint(buf, o, deltaStart); + o = writeVarint(buf, o, entries.length); + for (const s of entries) { + const n = Buffer.byteLength(s, "utf8"); + o = writeVarint(buf, o, n); + buf.write(s, o, "utf8"); + o += n; + } + } + for (const t of tables) { + o = writeString(buf, o, t.name); + o = writeVarint(buf, o, t.rowCount); + o = writeVarint(buf, o, t.columns.length); + for (const c of t.columns) { + o = writeString(buf, o, c.name); + buf.writeUInt8(c.type, o++); + } + for (const c of t.columns) o = writeColumn(buf, o, c, t.rowCount, colOpts); + } + if (o !== buf.length) throw new Error(`frame size mismatch: wrote ${o}, sized ${buf.length}`); + return buf; +} +``` + +In `columnWriter.ts` add `delta?: boolean` to `EncodeOpts`, and in the SYMBOL arms: + +```ts + // size + if (col.type === T.TYPE_SYMBOL) { + if (opts.delta) { + let n2 = 0; + for (const id of col.values as number[]) n2 += varintSize(id); + return n + n2; + } + // ...existing inline-dictionary sizing + } +``` + +```ts + case T.TYPE_SYMBOL: { + if (opts.delta) { + for (const id of col.values as number[]) o = writeVarint(buf, o, id); + return o; + } + // ...existing inline-dictionary writing + } +``` + +In `buffer.ts`, `symbol()` stores a global id when a dict is attached, otherwise the string: + +```ts + symbol(name: string, value: unknown): SenderBuffer { + return this.guard(() => { + const col = this.require().getOrCreateColumn(name, TYPE_SYMBOL); + if (col) col.values.push(this.dict ? this.dict.getOrAdd(String(value)) : String(value)); + return this; + }); + } +``` + +Add `private dict?: SymbolDict` and `attachDict(d: SymbolDict) { this.dict = d; }` to `QwpBuffer`, and pass `{ gorilla: false, dict: this.dict, confirmedMaxId: this.confirmedMaxId }` from `sealFrames`. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS. Plan 1's `frameEncoder.test.ts` needs its `encodeFrame([t])` calls updated to `encodeFrame([t], { gorilla: false })` — do that as part of this step. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/ src/qwp/buffer.ts test/qwp/ +git commit -m "feat(qwp): add delta symbol dictionary encoding" +``` + +--- + +### Task 8: Gorilla timestamps + +**Files:** +- Create: `src/qwp/protocol/bits.ts`, `src/qwp/protocol/gorilla.ts` +- Modify: `src/qwp/protocol/columnWriter.ts` +- Test: `test/qwp/gorilla.test.ts` + +**The trap (spec 6.3.2):** packing is LSB-first, so the prefix constants are **bit-reversed** relative to how they read. `'10'` is written as `writeBits(0b01, 2)`, `'110'` as `0b011`, `'1110'` as `0b0111`. Writing `0b10` for `'10'` produces plausible-but-wrong timestamps rather than a decode failure. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/gorilla.test.ts +import { describe, it, expect } from "vitest"; +import { BitWriter } from "../../src/qwp/protocol/bits"; +import { gorillaSize, encodeGorilla } from "../../src/qwp/protocol/gorilla"; + +describe("BitWriter", () => { + it("packs LSB-first within each byte", () => { + const w = new BitWriter(4); + w.writeBits(0b1, 1); + w.writeBits(0b0, 1); + w.writeBits(0b1, 1); + const out = w.finish(); + expect(out[0]).toBe(0b00000101); + }); +}); + +describe("gorilla", () => { + it("returns -1 when a delta-of-delta leaves int32", () => { + const ts = [0n, 1n, BigInt(2 ** 40)]; + expect(gorillaSize(ts)).toBe(-1); + }); + + it("sizes a constant-interval series as first two raw plus one bit per row", () => { + const ts = [1000n, 2000n, 3000n, 4000n]; + // 8 + 8 + ceil(2 bits / 8) = 17 + expect(gorillaSize(ts)).toBe(17); + }); + + it("emits the first two timestamps raw", () => { + const ts = [1000n, 2000n, 3000n]; + const b = encodeGorilla(ts); + expect(b.readBigInt64LE(0)).toBe(1000n); + expect(b.readBigInt64LE(8)).toBe(2000n); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/gorilla.test.ts` +Expected: FAIL — modules not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/protocol/bits.ts +import { Buffer } from "node:buffer"; + +/** LSB-first bit writer; trailing bits are zero-padded to a byte boundary. */ +export class BitWriter { + private readonly buf: Buffer; + private byteIndex = 0; + private bitIndex = 0; + + constructor(capacity: number) { + this.buf = Buffer.alloc(capacity); + } + + writeBits(value: number, count: number): void { + for (let i = 0; i < count; i++) { + if ((value >>> i) & 1) this.buf[this.byteIndex] |= 1 << this.bitIndex; + if (++this.bitIndex === 8) { + this.bitIndex = 0; + this.byteIndex++; + } + } + } + + finish(): Buffer { + const len = this.byteIndex + (this.bitIndex > 0 ? 1 : 0); + return this.buf.subarray(0, len); + } +} +``` + +```ts +// src/qwp/protocol/gorilla.ts +import { Buffer } from "node:buffer"; +import { BitWriter } from "./bits"; + +const INT32_MIN = -2147483648n; +const INT32_MAX = 2147483647n; + +function bitsRequired(dod: bigint): number { + if (dod === 0n) return 1; + if (dod >= -64n && dod <= 63n) return 9; + if (dod >= -256n && dod <= 255n) return 12; + if (dod >= -2048n && dod <= 2047n) return 16; + return 36; +} + +/** Encoded size in bytes, or -1 when a delta-of-delta leaves int32 range. */ +export function gorillaSize(ts: bigint[]): number { + if (ts.length === 0) return 0; + if (ts.length === 1) return 8; + if (ts.length === 2) return 16; + let prevTs = ts[1]; + let prevDelta = ts[1] - ts[0]; + let bits = 0; + for (let i = 2; i < ts.length; i++) { + const delta = ts[i] - prevTs; + const dod = delta - prevDelta; + if (dod < INT32_MIN || dod > INT32_MAX) return -1; + bits += bitsRequired(dod); + prevDelta = delta; + prevTs = ts[i]; + } + return 16 + Math.ceil(bits / 8); +} + +export function encodeGorilla(ts: bigint[]): Buffer { + const size = gorillaSize(ts); + if (size < 0) throw new Error("gorilla: delta-of-delta out of int32 range"); + const out = Buffer.alloc(size); + out.writeBigInt64LE(ts[0], 0); + if (ts.length === 1) return out; + out.writeBigInt64LE(ts[1], 8); + if (ts.length === 2) return out; + + const w = new BitWriter(size - 16); + let prevTs = ts[1]; + let prevDelta = ts[1] - ts[0]; + for (let i = 2; i < ts.length; i++) { + const delta = ts[i] - prevTs; + const dod = delta - prevDelta; + // Prefixes are BIT-REVERSED because packing is LSB-first (spec 6.3.2). + if (dod === 0n) { + w.writeBits(0b0, 1); + } else if (dod >= -64n && dod <= 63n) { + w.writeBits(0b01, 2); // logical '10' + w.writeBits(Number(dod & 0x7fn), 7); + } else if (dod >= -256n && dod <= 255n) { + w.writeBits(0b011, 3); // logical '110' + w.writeBits(Number(dod & 0x1ffn), 9); + } else if (dod >= -2048n && dod <= 2047n) { + w.writeBits(0b0111, 4); // logical '1110' + w.writeBits(Number(dod & 0xfffn), 12); + } else { + w.writeBits(0b1111, 4); // logical '1111' + w.writeBits(Number(dod & 0xffffffffn), 32); + } + prevDelta = delta; + prevTs = ts[i]; + } + w.finish().copy(out, 16); + return out; +} +``` + +In `columnWriter.ts`, timestamps become (spec 6.3.1) — note the encoding byte exists **only** when the gorilla flag is set, and is still emitted as `0x00` for columns of ≤ 2 values: + +```ts + case T.TYPE_TIMESTAMP: + case T.TYPE_TIMESTAMP_NANOS: { + const ts = col.values.map((v) => BigInt(v as bigint)); + if (!opts.gorilla) { + for (const v of ts) { + buf.writeBigInt64LE(v, o); + o += 8; + } + return o; + } + const size = ts.length > 2 ? gorillaSize(ts) : -1; + if (size > 0) { + buf.writeUInt8(T.ENCODING_GORILLA, o++); + encodeGorilla(ts).copy(buf, o); + return o + size; + } + buf.writeUInt8(T.ENCODING_UNCOMPRESSED, o++); + for (const v of ts) { + buf.writeBigInt64LE(v, o); + o += 8; + } + return o; + } +``` + +Mirror the same branching in `columnPayloadSize`. **DATE is excluded** — it stays in the fixed-width path and never carries an encoding byte. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/bits.ts src/qwp/protocol/gorilla.ts src/qwp/protocol/columnWriter.ts test/qwp/gorilla.test.ts +git commit -m "feat(qwp): add Gorilla timestamp encoding" +``` + +--- + +### Task 9: Cap-splitting and the commit frame + +**Files:** +- Modify: `src/qwp/buffer.ts`, `src/qwp/protocol/frameEncoder.ts` +- Test: `test/qwp/capSplit.test.ts` + +Spec 5.1 and 5.1.1. When the combined frame exceeds the cap, split **per table**; all but the last carry `FLAG_DEFER_COMMIT`. **Pre-flight every frame before publishing any.** The commit frame is `tableCount = 0`, no rows, flag cleared, and an empty delta built **by construction** from `[baseline+1 .. baseline]`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/capSplit.test.ts +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../src/qwp/buffer"; +import { encodeCommitFrame } from "../../src/qwp/protocol/frameEncoder"; +import { SymbolDict } from "../../src/qwp/protocol/symbolDict"; +import { FLAG_DEFER_COMMIT } from "../../src/qwp/protocol/constants"; + +describe("cap splitting", () => { + it("splits per table when the batch exceeds the cap", () => { + const b = new QwpBuffer(); + for (const t of ["a", "b", "c"]) { + b.table(t).intColumn("x", 1); + b.at(1n, "us"); + } + const frames = b.sealFrames(80); // small cap forces a split + expect(frames.length).toBe(3); + // All but the last defer the commit. + expect(frames[0].readUInt8(5) & FLAG_DEFER_COMMIT).toBe(FLAG_DEFER_COMMIT); + expect(frames[1].readUInt8(5) & FLAG_DEFER_COMMIT).toBe(FLAG_DEFER_COMMIT); + expect(frames[2].readUInt8(5) & FLAG_DEFER_COMMIT).toBe(0); + }); + + it("throws before publishing when one table cannot fit any split", () => { + const b = new QwpBuffer(); + b.table("wide").stringColumn("s", "x".repeat(500)); + b.at(1n, "us"); + expect(() => b.sealFrames(50)).toThrow(/cannot fit/i); + }); + + it("builds a commit frame with no tables and an empty delta", () => { + const dict = new SymbolDict(); + dict.getOrAdd("a"); + const f = encodeCommitFrame(dict, 0); + expect(f.readUInt16LE(6)).toBe(0); // tableCount + expect(f.readUInt8(5) & FLAG_DEFER_COMMIT).toBe(0); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/capSplit.test.ts` +Expected: FAIL — `encodeCommitFrame` not exported; `sealFrames` never splits. + +- [ ] **Step 3: Implement** + +Add to `frameEncoder.ts`: + +```ts +/** + * A commit carries no rows and MUST carry no symbols. The empty delta is built + * by construction from [baseline+1 .. baseline] — deriving the bound from batch + * state re-ships the whole dictionary in a frame no chunker covers (spec 5.1.1). + */ +export function encodeCommitFrame(dict: SymbolDict | undefined, baseline: number): Buffer { + return encodeFrame([], { gorilla: false, dict, confirmedMaxId: baseline, deferCommit: false }); +} +``` + +Because `entriesFrom(baseline + 1)` on a dictionary whose size is `baseline + 1` returns `[]`, the delta is empty by construction — no special-casing. + +Replace `sealFrames` in `buffer.ts`: + +```ts + sealFrames(maxBatchSize: number): Buffer[] { + const dirty = this.tables.filter((t) => t.rowCount > 0); + if (dirty.length === 0) return []; + + const opts = { gorilla: this.gorilla, dict: this.dict, confirmedMaxId: this.confirmedMaxId }; + const combined = encodeFrame(dirty, { ...opts, deferCommit: this.deferCommit }); + if (combined.length <= maxBatchSize) { + this.reset(); + return [combined]; + } + + // Pre-flight EVERY split frame before publishing any: discovering an + // oversized frame mid-publish strands the already-sent prefix and a later + // commit delivers a partial batch (spec 5.1). + const parts: Buffer[] = []; + for (let i = 0; i < dirty.length; i++) { + const isLast = i === dirty.length - 1; + const f = encodeFrame([dirty[i]], { + ...opts, + deferCommit: this.deferCommit ? true : !isLast, + }); + if (f.length > maxBatchSize) { + throw new Error( + `batch cannot fit the server cap however it is split ` + + `[table=${dirty[i].name}, frameSize=${f.length}, cap=${maxBatchSize}]`, + ); + } + parts.push(f); + } + this.reset(); + return parts; + } +``` + +Add `private gorilla = true;`, `private deferCommit = false;`, `private confirmedMaxId = -1;` to `QwpBuffer`, plus `setDeferCommit(on: boolean) { this.deferCommit = on; }`. + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run test/qwp/ && npx tsc --noEmit` +Expected: PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/buffer.ts src/qwp/protocol/frameEncoder.ts test/qwp/capSplit.test.ts +git commit -m "feat(qwp): add cap-splitting and the commit frame" +``` + +--- + +### Task 10: End-to-end across all types + +**Files:** +- Modify: `test/qwp/integration.test.ts` + +- [ ] **Step 1: Write the failing test** + +Append to the existing describe block from Plan 1: + +```ts + it("round-trips every supported column type", async () => { + const sender = await Sender.fromConfig( + `ws::addr=${container.getHost()}:${httpPort};`, + ); + await sender.connect(); + await sender + .table("qwp_types") + .symbol("sym", "A") + .stringColumn("str", "hello") + .booleanColumn("flag", true) + .intColumn("i", 42) + .floatColumn("d", 1.25) + .timestampColumn("ts2", 1_700_000_000_000_000n, "us") + .at(1_700_000_000_000_000n, "us"); + await sender.flush(); + await sender.close(); + + let rows: any[] = []; + for (let i = 0; i < 60; i++) { + const r = await query("select sym, str, flag, i, d from qwp_types"); + rows = r.dataset ?? []; + if (rows.length > 0) break; + await new Promise((r) => setTimeout(r, 500)); + } + expect(rows[0][0]).toBe("A"); + expect(rows[0][1]).toBe("hello"); + expect(rows[0][2]).toBe(true); + expect(rows[0][3]).toBe(42); + expect(rows[0][4]).toBeCloseTo(1.25, 5); + }, 180_000); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/integration.test.ts` +Expected: FAIL — `booleanColumn` still throws "not supported" until Task 2 lands, then passes. + +- [ ] **Step 3: Wire the remaining setters in `buffer.ts`** + +```ts + booleanColumn(name: string, value: boolean): SenderBuffer { + return this.guard(() => { + const col = this.require().getOrCreateColumn(name, TYPE_BOOLEAN); + if (col) col.values.push(value); + return this; + }); + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `npx vitest run && npx tsc --noEmit && npx eslint src/**` +Expected: all green. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/buffer.ts test/qwp/integration.test.ts +git commit -m "test(qwp): end-to-end coverage for all supported column types" +``` + +--- + +## Self-Review + +**1. Spec coverage.** Remaining scalars (Task 2 — 6.3). VARCHAR/BINARY (Task 3). Arrays/decimals/geohash with locks (Task 4 — 6.5.3). Row rollback (Task 5 — 4.1.1). Symbol dict, both modes (Tasks 6, 7 — 5.2, 6.2). Gorilla with bit-reversed prefixes (Task 8 — 6.3.2). Cap-split + commit frame (Task 9 — 5.1, 5.1.1). Multi-frame contract (Task 1 — 3.1). + +**Deferred to Plan 3, not dropped:** the delta→full-dict runtime fallback (5.2) needs the `.symbol-dict` file from Plan 4, so `QwpBuffer` starts in full-dict mode unless a dict is attached; `DICTIONARY_GAP` handling needs the ACK path. + +**2. Placeholder scan.** None. Every step carries code. + +**3. Type consistency.** `EncodeOpts { gorilla, delta? }` (Task 2) gains `delta` in Task 7 and is used in Task 8. `FrameOpts` (Task 7) is used by `encodeCommitFrame` (Task 9). `SymbolDict.entriesFrom` (Task 6) is called in Task 7 and relied on for the empty-by-construction commit delta in Task 9. `sealFrames(maxBatchSize)` (Task 1) is re-implemented in Task 9 with the same signature. + +**Known churn, stated:** Task 7 changes `encodeFrame`'s arity, so Plan 1's `frameEncoder.test.ts` must be updated in that task's Step 4. Task 2 moves `columnPayloadSize`/`writeColumn` out of `frameEncoder.ts`; Plan 1's tests import from `frameEncoder` only via `encodeFrame`, so they are unaffected. diff --git a/docs/superpowers/plans/2026-08-07-qwp-plan-3-errors-and-failover.md b/docs/superpowers/plans/2026-08-07-qwp-plan-3-errors-and-failover.md new file mode 100644 index 0000000..af93cf4 --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-qwp-plan-3-errors-and-failover.md @@ -0,0 +1,1364 @@ +# QWP Plan 3 — Errors, Reconnect and Failover (spec PRs 9–11) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Read the server's responses, classify rejections correctly, survive a disconnect, and walk a list of endpoints — so a QWP sender keeps running across restarts, failovers and read-only windows. + +**Architecture:** Adds a response decoder and an error-policy layer to `src/qwp/`, plus an endpoint list and a state-ranked host tracker. The send loop gains a connection lifecycle: connect → send → observe ACKs → on failure, classify, reconnect, re-register the symbol dictionary. + +**Tech Stack:** TypeScript, Node ≥ 20. vitest + testcontainers + an in-process mock QWP server. + +**Prerequisites:** Plans 1 and 2 merged. Consumes: `QwpWebSocket`, `QwpTransport`, `QwpBuffer`, `SymbolDict`, `encodeFrame`/`encodeCommitFrame`, the constants module. + +**Source of truth:** `docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md`. + +## Global Constraints + +- **No new runtime dependencies.** +- **Node 20 floor.** +- **Options stay `undefined` until set** (spec 9.1.2) — the connect-mode derivation in Task 9 depends on distinguishing "unset" from "set to the default". +- Existing tests stay green: `npx vitest run && npx tsc --noEmit && npx eslint src/**`. + +## Sequencing correction — read this before starting + +The spec's PR 9 lists "replay" alongside ACK handling. **Replay is not implementable in this plan.** Replay resends frames from `ackedFsn + 1`, which requires the retention ring that Plan 4 builds — spec 12 records that "between PR 3 and PR 12 there is no retention". So this plan: + +- **does** implement: response decoding, ACK→FSN correlation, error categories and policies, the poison detector, reconnect with backoff, dictionary catch-up, the endpoint list, and host-state ranking; +- **does not** implement: replaying unacked frames. Until Plan 4, a disconnect loses in-flight frames and that must be surfaced through the error handler rather than hidden. + +Task 4 makes that loss explicit rather than silent. + +## File Structure + +| File | Responsibility | +|---|---| +| `src/qwp/protocol/response.ts` | Decode OK / DURABLE_ACK / error frames | +| `src/qwp/errors.ts` | `SenderErrorCategory`, `SenderErrorPolicy`, `defaultPolicyFor`, `SenderError` | +| `src/qwp/ackTracker.ts` | `fsnAtZero` / `nextWireSeq` translation and clamping | +| `src/qwp/poison.ts` | Strike + dwell escalation | +| `src/qwp/endpoints.ts` | `addr` list grammar, IPv6-aware | +| `src/qwp/hostTracker.ts` | State-ranked, round-based endpoint selection | +| `src/qwp/dispatcher.ts` | Bounded drop-oldest notification inbox | +| `src/qwp/transport.ts` | **modify** — lifecycle, reconnect, catch-up | +| `test/qwp/mockServer.ts` | Reusable in-process QWP server for tests | + +--- + +### Task 1: Response decoder + +**Files:** +- Create: `src/qwp/protocol/response.ts` +- Test: `test/qwp/response.test.ts` + +**Interfaces:** +- Produces: `STATUS` map, `decodeResponse(payload: Buffer): QwpResponse`, `type QwpResponse = { status: number; sequence: number; tables: {name: string; seqTxn: bigint}[]; errorMessage?: string }`. + +Wire layouts from spec 6.6. Note `DURABLE_ACK` has **no** sequence field. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/response.test.ts +import { describe, it, expect } from "vitest"; +import { decodeResponse, STATUS } from "../../src/qwp/protocol/response"; + +function ok(seq: number, tables: [string, bigint][]): Buffer { + const parts: Buffer[] = []; + const head = Buffer.alloc(11); + head.writeUInt8(STATUS.OK, 0); + head.writeBigUInt64LE(BigInt(seq), 1); + head.writeUInt16LE(tables.length, 9); + parts.push(head); + for (const [name, txn] of tables) { + const n = Buffer.byteLength(name, "utf8"); + const e = Buffer.alloc(2 + n + 8); + e.writeUInt16LE(n, 0); + e.write(name, 2, "utf8"); + e.writeBigInt64LE(txn, 2 + n); + parts.push(e); + } + return Buffer.concat(parts); +} + +describe("decodeResponse", () => { + it("decodes an OK with per-table seqTxn", () => { + const r = decodeResponse(ok(7, [["trades", 42n]])); + expect(r.status).toBe(STATUS.OK); + expect(r.sequence).toBe(7); + expect(r.tables).toEqual([{ name: "trades", seqTxn: 42n }]); + }); + + it("decodes an error with its message", () => { + const msg = "boom"; + const b = Buffer.alloc(11 + msg.length); + b.writeUInt8(STATUS.WRITE_ERROR, 0); + b.writeBigUInt64LE(3n, 1); + b.writeUInt16LE(msg.length, 9); + b.write(msg, 11, "utf8"); + const r = decodeResponse(b); + expect(r.status).toBe(STATUS.WRITE_ERROR); + expect(r.errorMessage).toBe("boom"); + }); + + it("decodes DURABLE_ACK, which carries no sequence", () => { + const b = Buffer.alloc(3); + b.writeUInt8(STATUS.DURABLE_ACK, 0); + b.writeUInt16LE(0, 1); + expect(decodeResponse(b).status).toBe(STATUS.DURABLE_ACK); + }); + + it("rejects a truncated payload", () => { + expect(() => decodeResponse(Buffer.alloc(2))).toThrow(/invalid|truncated/i); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `npx vitest run test/qwp/response.test.ts` → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/protocol/response.ts +import { Buffer } from "node:buffer"; + +export const STATUS = { + OK: 0x00, + DURABLE_ACK: 0x02, + SCHEMA_MISMATCH: 0x03, + PARSE_ERROR: 0x05, + INTERNAL_ERROR: 0x06, + SECURITY_ERROR: 0x08, + WRITE_ERROR: 0x09, + CANCELLED: 0x0a, + LIMIT_EXCEEDED: 0x0b, + NOT_WRITABLE: 0x0c, + DICTIONARY_GAP: 0x0d, +} as const; + +export const MAX_ERROR_MESSAGE_LENGTH = 1024; + +export interface QwpResponse { + status: number; + sequence: number; + tables: { name: string; seqTxn: bigint }[]; + errorMessage?: string; +} + +export function decodeResponse(payload: Buffer): QwpResponse { + if (payload.length < 3) throw new Error("invalid QWP response: truncated"); + const status = payload.readUInt8(0); + + if (status === STATUS.DURABLE_ACK) { + const count = payload.readUInt16LE(1); + return { status, sequence: -1, tables: readTables(payload, 3, count) }; + } + + if (payload.length < 11) throw new Error("invalid QWP response: truncated"); + const sequence = Number(payload.readBigUInt64LE(1)); + + if (status === STATUS.OK) { + const count = payload.readUInt16LE(9); + return { status, sequence, tables: readTables(payload, 11, count) }; + } + + const len = payload.readUInt16LE(9); + if (len > MAX_ERROR_MESSAGE_LENGTH) throw new Error("invalid QWP response: error message too long"); + return { + status, + sequence, + tables: [], + errorMessage: payload.subarray(11, 11 + len).toString("utf8"), + }; +} + +function readTables(buf: Buffer, offset: number, count: number) { + const out: { name: string; seqTxn: bigint }[] = []; + let o = offset; + for (let i = 0; i < count; i++) { + const n = buf.readUInt16LE(o); + o += 2; + const name = buf.subarray(o, o + n).toString("utf8"); + o += n; + out.push({ name, seqTxn: buf.readBigInt64LE(o) }); + o += 8; + } + return out; +} +``` + +- [ ] **Step 4: Run test** → `npx vitest run test/qwp/response.test.ts` → PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/protocol/response.ts test/qwp/response.test.ts +git commit -m "feat(qwp): decode server response frames" +``` + +--- + +### Task 2: Error categories and default policy + +**Files:** +- Create: `src/qwp/errors.ts` +- Test: `test/qwp/errors.test.ts` + +Spec 7.1–7.3. Ten categories, four policies. Three mappings are **forced** and ignore any override. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/errors.test.ts +import { describe, it, expect } from "vitest"; +import { Category, Policy, classify, defaultPolicyFor } from "../../src/qwp/errors"; +import { STATUS } from "../../src/qwp/protocol/response"; + +describe("error classification", () => { + it("maps wire statuses to categories", () => { + expect(classify(STATUS.SCHEMA_MISMATCH)).toBe(Category.SCHEMA_MISMATCH); + expect(classify(STATUS.DICTIONARY_GAP)).toBe(Category.DICTIONARY_GAP); + expect(classify(0x7f)).toBe(Category.UNKNOWN); + }); + + it("fails OPEN on an unknown status", () => { + expect(defaultPolicyFor(Category.UNKNOWN)).toBe(Policy.RETRIABLE); + }); + + it("treats deterministic rejections as terminal", () => { + for (const c of [Category.SCHEMA_MISMATCH, Category.PARSE_ERROR, Category.SECURITY_ERROR]) { + expect(defaultPolicyFor(c)).toBe(Policy.TERMINAL); + } + }); + + it("routes DICTIONARY_GAP to retriable, not terminal", () => { + expect(defaultPolicyFor(Category.DICTIONARY_GAP)).toBe(Policy.RETRIABLE); + }); + + it("maps NOT_WRITABLE to RETRIABLE_OTHER and DATA_LOSS to ABANDONED", () => { + expect(defaultPolicyFor(Category.NOT_WRITABLE)).toBe(Policy.RETRIABLE_OTHER); + expect(defaultPolicyFor(Category.DATA_LOSS)).toBe(Policy.ABANDONED); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/errors.ts +import { STATUS } from "./protocol/response"; + +export enum Category { + SCHEMA_MISMATCH = "SCHEMA_MISMATCH", + PARSE_ERROR = "PARSE_ERROR", + INTERNAL_ERROR = "INTERNAL_ERROR", + SECURITY_ERROR = "SECURITY_ERROR", + WRITE_ERROR = "WRITE_ERROR", + NOT_WRITABLE = "NOT_WRITABLE", + DICTIONARY_GAP = "DICTIONARY_GAP", + PROTOCOL_VIOLATION = "PROTOCOL_VIOLATION", + DATA_LOSS = "DATA_LOSS", + UNKNOWN = "UNKNOWN", +} + +export enum Policy { + RETRIABLE = "RETRIABLE", + RETRIABLE_OTHER = "RETRIABLE_OTHER", + TERMINAL = "TERMINAL", + ABANDONED = "ABANDONED", +} + +export class SenderError extends Error { + constructor( + readonly category: Category, + readonly policy: Policy, + message: string, + readonly serverStatus = -1, + readonly fromFsn = -1, + readonly toFsn = -1, + readonly quarantinedPath?: string, + ) { + super(message); + this.name = "SenderError"; + } +} + +export function classify(status: number): Category { + switch (status) { + case STATUS.SCHEMA_MISMATCH: return Category.SCHEMA_MISMATCH; + case STATUS.PARSE_ERROR: return Category.PARSE_ERROR; + case STATUS.INTERNAL_ERROR: return Category.INTERNAL_ERROR; + case STATUS.SECURITY_ERROR: return Category.SECURITY_ERROR; + case STATUS.WRITE_ERROR: return Category.WRITE_ERROR; + case STATUS.NOT_WRITABLE: return Category.NOT_WRITABLE; + case STATUS.DICTIONARY_GAP: return Category.DICTIONARY_GAP; + default: return Category.UNKNOWN; + } +} + +/** + * There is no drop policy. UNKNOWN fails OPEN so a status byte from a newer + * server degrades to a retry rather than a dead sender (spec 7.3). + */ +export function defaultPolicyFor(c: Category): Policy { + switch (c) { + case Category.WRITE_ERROR: + case Category.INTERNAL_ERROR: + case Category.DICTIONARY_GAP: + case Category.UNKNOWN: + return Policy.RETRIABLE; + case Category.NOT_WRITABLE: + return Policy.RETRIABLE_OTHER; + case Category.DATA_LOSS: + return Policy.ABANDONED; + default: + return Policy.TERMINAL; + } +} +``` + +- [ ] **Step 4: Run test** → PASS, 5 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/errors.ts test/qwp/errors.test.ts +git commit -m "feat(qwp): add error categories and default policy mapping" +``` + +--- + +### Task 3: ACK→FSN correlation + +**Files:** +- Create: `src/qwp/ackTracker.ts` +- Test: `test/qwp/ackTracker.test.ts` + +**Spec 6.6.1 — the highest-risk item in this plan.** The wire `seq` is **connection-scoped** and restarts at 0 on every reconnect; FSNs are monotonic for the life of the log. `ackedFsn = fsnAtZero + seq`, clamped to what was actually sent. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/ackTracker.test.ts +import { describe, it, expect } from "vitest"; +import { AckTracker } from "../../src/qwp/ackTracker"; + +describe("AckTracker", () => { + it("translates a connection-scoped seq into an FSN", () => { + const t = new AckTracker(); + t.onConnected(100); // replay resumes at FSN 100 + t.onFrameSent(); + t.onFrameSent(); + expect(t.onAck(1)).toBe(101); + }); + + it("does NOT reset the FSN when the wire seq restarts", () => { + const t = new AckTracker(); + t.onConnected(0); + t.onFrameSent(); + expect(t.onAck(0)).toBe(0); + // reconnect: wire seq restarts at 0, FSNs continue from 1 + t.onConnected(1); + t.onFrameSent(); + expect(t.onAck(0)).toBe(1); + }); + + it("clamps an ACK beyond what was sent", () => { + const t = new AckTracker(); + t.onConnected(0); + t.onFrameSent(); // highest wire seq is 0 + expect(t.onAck(99)).toBe(0); + }); + + it("ignores an ACK arriving before any send", () => { + const t = new AckTracker(); + t.onConnected(0); + expect(t.onAck(0)).toBeNull(); + }); + + it("never moves the acked watermark backwards", () => { + const t = new AckTracker(); + t.onConnected(0); + t.onFrameSent(); + t.onFrameSent(); + expect(t.onAck(1)).toBe(1); + expect(t.onAck(0)).toBe(1); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/ackTracker.ts + +/** + * Bridges the connection-scoped wire sequence to the log-scoped FSN. + * Storing a raw seq as an FSN works until the first reconnect and then trims + * from near the start of the log, discarding unacked data (spec 6.6.1). + */ +export class AckTracker { + private fsnAtZero = 0; + private nextWireSeq = 0; + private ackedFsn = -1; + + /** Call on every successful connect, with the FSN replay resumes at. */ + onConnected(replayStartFsn: number): void { + this.fsnAtZero = replayStartFsn; + this.nextWireSeq = 0; + } + + onFrameSent(): void { + this.nextWireSeq++; + } + + /** Returns the new acked FSN, or null when the ACK is not applicable. */ + onAck(wireSeq: number): number | null { + const highestSent = this.nextWireSeq - 1; + if (highestSent < 0) return null; // ACK before any send + const capped = Math.max(0, Math.min(wireSeq, highestSent)); + const fsn = this.fsnAtZero + capped; + if (fsn > this.ackedFsn) this.ackedFsn = fsn; + return this.ackedFsn; + } + + get acked(): number { + return this.ackedFsn; + } +} +``` + +- [ ] **Step 4: Run test** → PASS, 5 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/ackTracker.ts test/qwp/ackTracker.test.ts +git commit -m "feat(qwp): correlate connection-scoped ACK sequences to FSNs" +``` + +--- + +### Task 4: Mock QWP server, and surfacing in-flight loss + +**Files:** +- Create: `test/qwp/mockServer.ts` +- Modify: `src/qwp/transport.ts` +- Test: `test/qwp/transport.acks.test.ts` + +Until Plan 4 there is no retention, so a disconnect with frames in flight **loses them**. That must be reported, not swallowed. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/mockServer.ts +import { createServer, Server, Socket } from "node:net"; +import { createHash } from "node:crypto"; +import { FrameParser, encodeClientFrame, OPCODE } from "../../src/qwp/ws/frame"; +import { STATUS } from "../../src/qwp/protocol/response"; + +export interface MockOptions { + /** Return a status per received frame; OK by default. */ + statusFor?: (frameIndex: number) => number; + /** Drop the connection after N frames. */ + dropAfter?: number; + upgradeStatus?: number; + upgradeHeaders?: string; +} + +export function okResponse(seq: number): Buffer { + const b = Buffer.alloc(11); + b.writeUInt8(STATUS.OK, 0); + b.writeBigUInt64LE(BigInt(seq), 1); + b.writeUInt16LE(0, 9); + return b; +} + +export function errorResponse(status: number, seq: number, msg: string): Buffer { + const b = Buffer.alloc(11 + msg.length); + b.writeUInt8(status, 0); + b.writeBigUInt64LE(BigInt(seq), 1); + b.writeUInt16LE(msg.length, 9); + b.write(msg, 11, "utf8"); + return b; +} + +export class MockQwpServer { + private server?: Server; + readonly frames: Buffer[] = []; + + async start(opts: MockOptions = {}): Promise { + return new Promise((resolve) => { + this.server = createServer((sock: Socket) => this.onConn(sock, opts)); + this.server.listen(0, "127.0.0.1", () => + resolve((this.server!.address() as any).port), + ); + }); + } + + async stop(): Promise { + await new Promise((r) => this.server?.close(() => r())); + } + + private onConn(sock: Socket, opts: MockOptions): void { + let handshaken = false; + let seq = 0; + const parser = new FrameParser(); + sock.on("error", () => undefined); + sock.on("data", (chunk: Buffer) => { + if (!handshaken) { + const status = opts.upgradeStatus ?? 101; + if (status !== 101) { + sock.write(`HTTP/1.1 ${status} X\r\n${opts.upgradeHeaders ?? ""}\r\n`); + sock.end(); + return; + } + const key = /Sec-WebSocket-Key: (.+)\r\n/.exec(chunk.toString("ascii"))![1]; + const accept = createHash("sha1") + .update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11", "ascii") + .digest("base64"); + sock.write( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n" + + `Connection: Upgrade\r\nSec-WebSocket-Accept: ${accept}\r\n` + + "X-QWP-Version: 1\r\nX-QWP-Max-Batch-Size: 1048576\r\n\r\n", + ); + handshaken = true; + return; + } + parser.push(chunk); + for (let m = parser.next(); m; m = parser.next()) { + if (m.opcode !== OPCODE.BINARY) continue; + const idx = this.frames.length; + this.frames.push(m.payload); + if (opts.dropAfter !== undefined && idx + 1 >= opts.dropAfter) { + sock.destroy(); + return; + } + const status = opts.statusFor ? opts.statusFor(idx) : STATUS.OK; + const body = + status === STATUS.OK ? okResponse(seq++) : errorResponse(status, seq++, "mock"); + sock.write(encodeClientFrame(OPCODE.BINARY, body)); + } + }); + } +} +``` + +```ts +// test/qwp/transport.acks.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { MockQwpServer } from "./mockServer"; +import { QwpTransport } from "../../src/qwp/transport"; +import { SenderOptions } from "../../src/options"; +import { STATUS } from "../../src/qwp/protocol/response"; +import { Category } from "../../src/qwp/errors"; + +let mock: MockQwpServer | undefined; +afterEach(async () => await mock?.stop()); + +async function connected(opts = {}) { + mock = new MockQwpServer(); + const port = await mock.start(opts); + const t = new QwpTransport(new SenderOptions(`ws::addr=127.0.0.1:${port};`)); + await t.connect(); + return t; +} + +describe("QwpTransport ack handling", () => { + it("advances the acked FSN on OK", async () => { + const t = await connected(); + await t.sendFrames([Buffer.from("QWP1----------")]); + await new Promise((r) => setTimeout(r, 100)); + expect(t.ackedFsn).toBe(0); + await t.close(); + }); + + it("reports a terminal category on a deterministic NACK", async () => { + const errors: any[] = []; + const t = await connected({ statusFor: () => STATUS.PARSE_ERROR }); + t.onError((e) => errors.push(e)); + await t.sendFrames([Buffer.from("QWP1----------")]); + await new Promise((r) => setTimeout(r, 100)); + expect(errors[0].category).toBe(Category.PARSE_ERROR); + await t.close(); + }); + + it("reports in-flight loss when the connection drops (no retention yet)", async () => { + const errors: any[] = []; + const t = await connected({ dropAfter: 1 }); + t.onError((e) => errors.push(e)); + await t.sendFrames([Buffer.from("QWP1----------")]); + await new Promise((r) => setTimeout(r, 200)); + expect(errors.some((e) => e.category === Category.DATA_LOSS)).toBe(true); + await t.close(); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, `t.ackedFsn` / `t.onError` undefined. + +- [ ] **Step 3: Implement** — extend `src/qwp/transport.ts`: + +```ts + private readonly acks = new AckTracker(); + private errorHandler?: (e: SenderError) => void; + private inFlight = 0; + + onError(h: (e: SenderError) => void): void { + this.errorHandler = h; + } + + get ackedFsn(): number { + return this.acks.acked; + } + + private emit(e: SenderError): void { + try { + this.errorHandler?.(e); + } catch { + /* a handler must never break the sender (spec 4.2) */ + } + } + + private onResponse(payload: Buffer): void { + const r = decodeResponse(payload); + if (r.status === STATUS.OK) { + this.inFlight = Math.max(0, this.inFlight - 1); + this.acks.onAck(r.sequence); + return; + } + if (r.status === STATUS.DURABLE_ACK) return; + const category = classify(r.status); + this.emit( + new SenderError( + category, + defaultPolicyFor(category), + r.errorMessage ?? `server rejected frame [status=0x${r.status.toString(16)}]`, + r.status, + ), + ); + } + + private onDisconnected(): void { + if (this.inFlight > 0) { + // No retention until Plan 4: these frames are gone. Say so. + this.emit( + new SenderError( + Category.DATA_LOSS, + Policy.ABANDONED, + `connection lost with ${this.inFlight} frame(s) in flight and no retention configured`, + ), + ); + this.inFlight = 0; + } + } +``` + +Wire `onResponse`/`onDisconnected` into `QwpWebSocket` via two new callbacks (`onBinary`, `onClose`) passed to `QwpWebSocket.connect`, and increment `inFlight`/`acks.onFrameSent()` inside `sendFrames`. Call `this.acks.onConnected(0)` at the end of `connect()`. + +- [ ] **Step 4: Run test** → PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/transport.ts src/qwp/ws/socket.ts test/qwp/mockServer.ts test/qwp/transport.acks.test.ts +git commit -m "feat(qwp): handle ACK and NACK responses, surface in-flight loss" +``` + +--- + +### Task 5: Poison-frame detector + +**Files:** +- Create: `src/qwp/poison.ts` +- Test: `test/qwp/poison.test.ts` + +**Spec 7.4 — escalation needs BOTH conditions.** Four strikes alone is not enough; the suspect must also have stayed poisoned for `poison_min_escalation_window_millis` (default 5000). Count-only escalation turns a brief outage into a producer-fatal terminal. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/poison.test.ts +import { describe, it, expect } from "vitest"; +import { PoisonDetector } from "../../src/qwp/poison"; + +describe("PoisonDetector", () => { + it("does NOT escalate on strikes alone inside the dwell window", () => { + let now = 1000; + const d = new PoisonDetector(4, 5000, () => now); + for (let i = 0; i < 6; i++) { + now += 100; + expect(d.strike(7)).toBe(false); + } + }); + + it("escalates once both the count and the dwell are satisfied", () => { + let now = 1000; + const d = new PoisonDetector(4, 5000, () => now); + d.strike(7); + for (let i = 0; i < 3; i++) { + now += 2000; + d.strike(7); + } + now += 1; + expect(d.strike(7)).toBe(true); + }); + + it("resets only on acceptance at or beyond the suspect frame", () => { + let now = 1000; + const d = new PoisonDetector(4, 0, () => now); + d.strike(7); + d.accept(5); // behind the suspect: must NOT launder the count + d.strike(7); + d.strike(7); + expect(d.strike(7)).toBe(true); + + const d2 = new PoisonDetector(4, 0, () => now); + d2.strike(7); + d2.accept(7); // at the suspect: clears + expect(d2.strike(7)).toBe(false); + }); + + it("a different frame resets the sequence", () => { + let now = 1000; + const d = new PoisonDetector(4, 0, () => now); + d.strike(7); + d.strike(7); + d.strike(8); + expect(d.strike(8)).toBe(false); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/poison.ts + +/** + * Escalation requires a strike count AND a wall-clock dwell (spec 7.4). + * A count alone false-positives a brief outage into a producer-fatal terminal, + * because with pacing four strikes can accrue in well under a second. + */ +export class PoisonDetector { + private suspectFsn = -1; + private strikes = 0; + private firstStrikeAt = 0; + + constructor( + private readonly maxStrikes: number, + private readonly minWindowMillis: number, + private readonly now: () => number = () => Date.now(), + ) {} + + /** Returns true when the frame should escalate to PROTOCOL_VIOLATION. */ + strike(fsn: number): boolean { + if (fsn !== this.suspectFsn) { + this.suspectFsn = fsn; + this.strikes = 0; + this.firstStrikeAt = this.now(); + } + this.strikes++; + const dwell = this.now() - this.firstStrikeAt; + return this.strikes >= this.maxStrikes && dwell >= this.minWindowMillis; + } + + /** Only acceptance AT OR BEYOND the suspect clears it. */ + accept(ackedFsn: number): void { + if (this.suspectFsn >= 0 && ackedFsn >= this.suspectFsn) { + this.suspectFsn = -1; + this.strikes = 0; + } + } +} +``` + +- [ ] **Step 4: Run test** → PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/poison.ts test/qwp/poison.test.ts +git commit -m "feat(qwp): add poison-frame detector with strike and dwell conditions" +``` + +--- + +### Task 6: `addr` list grammar + +**Files:** +- Create: `src/qwp/endpoints.ts` +- Modify: `src/options.ts` +- Test: `test/qwp/endpoints.test.ts` + +Spec 1.2. Comma-separated, IPv6-aware, duplicates rejected on `(host, port)`. **A custom port on IPv6 requires brackets** — an unbracketed multi-colon entry is a bare IPv6 host on the default port. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/endpoints.test.ts +import { describe, it, expect } from "vitest"; +import { parseAddrList } from "../../src/qwp/endpoints"; + +describe("parseAddrList", () => { + it("parses host and host:port", () => { + expect(parseAddrList("a,b:1234", 9000)).toEqual([ + { host: "a", port: 9000 }, + { host: "b", port: 1234 }, + ]); + }); + + it("parses bracketed IPv6 with and without a port", () => { + expect(parseAddrList("[::1]:9001,[fe80::1]", 9000)).toEqual([ + { host: "::1", port: 9001 }, + { host: "fe80::1", port: 9000 }, + ]); + }); + + it("treats an unbracketed multi-colon entry as bare IPv6 on the default port", () => { + expect(parseAddrList("fe80::1", 9000)).toEqual([{ host: "fe80::1", port: 9000 }]); + }); + + it("rejects duplicates on (host, port) but allows the same host twice on different ports", () => { + expect(() => parseAddrList("a:1,a:1", 9000)).toThrow(/duplicate/i); + expect(parseAddrList("a:1,a:2", 9000).length).toBe(2); + }); + + it("rejects a missing bracket and an empty host", () => { + expect(() => parseAddrList("[::1:9000", 9000)).toThrow(/closing/i); + expect(() => parseAddrList(":9000", 9000)).toThrow(/empty host/i); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/endpoints.ts +export interface Endpoint { + host: string; + port: number; +} + +export function parseAddrList(addr: string, defaultPort: number): Endpoint[] { + const out: Endpoint[] = []; + const seen = new Set(); + for (const raw of addr.split(",")) { + const entry = raw.trim(); + if (!entry) continue; + let host: string; + let port: number; + + if (entry[0] === "[") { + const close = entry.indexOf("]"); + if (close < 0) throw new Error(`missing closing ']' in IPv6 addr entry: ${entry}`); + host = entry.slice(1, close); + if (close === entry.length - 1) { + port = defaultPort; + } else if (entry[close + 1] !== ":") { + throw new Error(`expected ':' after ']' in IPv6 addr entry: ${entry}`); + } else { + port = parsePort(entry.slice(close + 2), entry); + } + } else if (entry.indexOf(":") !== entry.lastIndexOf(":")) { + // Unbracketed multi-colon: bare IPv6, default port. A custom port needs brackets. + host = entry; + port = defaultPort; + } else { + const colon = entry.indexOf(":"); + if (colon < 0) { + host = entry; + port = defaultPort; + } else { + host = entry.slice(0, colon).trim(); + port = parsePort(entry.slice(colon + 1), entry); + } + } + + if (!host) throw new Error(`empty host in addr entry: ${entry}`); + const key = `${port}/${host}`; + if (seen.has(key)) throw new Error(`duplicate addr entry: ${entry}`); + seen.add(key); + out.push({ host, port }); + } + if (out.length === 0) throw new Error("addr is missing"); + return out; +} + +function parsePort(s: string, entry: string): number { + const p = Number.parseInt(s, 10); + if (!Number.isInteger(p) || p < 1 || p > 65535) { + throw new Error(`invalid port in addr entry: ${entry}`); + } + return p; +} +``` + +In `src/options.ts`, for `ws`/`wss` store the parsed list on the options object as `endpoints` and keep `host`/`port` pointing at the first entry so existing code paths still work. + +- [ ] **Step 4: Run test** → `npx vitest run test/qwp/ && npx vitest run` → PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/endpoints.ts src/options.ts test/qwp/endpoints.test.ts +git commit -m "feat(qwp): parse multi-host addr lists with IPv6 support" +``` + +--- + +### Task 7: Host health tracker + +**Files:** +- Create: `src/qwp/hostTracker.ts` +- Test: `test/qwp/hostTracker.test.ts` + +Spec 1.2. **State-only ranking** — the ingest sender is zone-blind, so do not implement zone tiers. Rounds via `pickNext`/`beginRound`. Background drainers (Plan 4) will need `newCursor()`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/hostTracker.test.ts +import { describe, it, expect } from "vitest"; +import { HostTracker, HostState } from "../../src/qwp/hostTracker"; + +describe("HostTracker", () => { + it("prefers a known-good host over an untried one", () => { + const t = new HostTracker(3); + t.record(2, HostState.HEALTHY); + expect(t.pickNext()).toBe(2); + }); + + it("ranks HEALTHY > UNKNOWN > TRANSIENT_REJECT > TRANSPORT_ERROR > TOPOLOGY_REJECT", () => { + const t = new HostTracker(4); + t.record(0, HostState.TOPOLOGY_REJECT); + t.record(1, HostState.TRANSPORT_ERROR); + t.record(2, HostState.TRANSIENT_REJECT); + // index 3 stays UNKNOWN + expect(t.pickNext()).toBe(3); + expect(t.pickNext()).toBe(2); + expect(t.pickNext()).toBe(1); + expect(t.pickNext()).toBe(0); + }); + + it("exhausts a round and restarts on beginRound", () => { + const t = new HostTracker(2); + t.pickNext(); + t.pickNext(); + expect(t.pickNext()).toBeNull(); + expect(t.isRoundExhausted()).toBe(true); + t.beginRound(); + expect(t.pickNext()).not.toBeNull(); + }); + + it("a private cursor does not consume the shared round", () => { + const t = new HostTracker(2); + const c = t.newCursor(); + c.pickNext(); + c.pickNext(); + expect(t.isRoundExhausted()).toBe(false); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/hostTracker.ts + +export enum HostState { + HEALTHY = 0, + UNKNOWN = 1, + TRANSIENT_REJECT = 2, + TRANSPORT_ERROR = 3, + TOPOLOGY_REJECT = 4, +} + +/** + * State-ranked, round-based endpoint selection. The ingest sender is + * zone-blind, so ranking is state-only (spec 1.2) — do not add zone tiers. + */ +export class HostTracker { + private readonly states: HostState[]; + private attempted: boolean[]; + + constructor(private readonly hostCount: number) { + if (hostCount <= 0) throw new Error("hostCount must be > 0"); + this.states = new Array(hostCount).fill(HostState.UNKNOWN); + this.attempted = new Array(hostCount).fill(false); + } + + record(index: number, state: HostState): void { + this.states[index] = state; + } + + private best(attempted: boolean[]): number | null { + let bestIdx: number | null = null; + for (let i = 0; i < this.hostCount; i++) { + if (attempted[i]) continue; + if (bestIdx === null || this.states[i] < this.states[bestIdx]) bestIdx = i; + } + return bestIdx; + } + + pickNext(): number | null { + const i = this.best(this.attempted); + if (i === null) return null; + this.attempted[i] = true; + return i; + } + + isRoundExhausted(): boolean { + return this.attempted.every(Boolean); + } + + beginRound(): void { + this.attempted = new Array(this.hostCount).fill(false); + } + + /** + * A walker-local cursor. Background drainers MUST use this: sharing the + * round lets a drainer steal endpoints from the foreground sweep, which + * presents as unexplained ALL_ENDPOINTS_UNREACHABLE (spec 1.2). + */ + newCursor(): { pickNext(): number | null } { + const local = new Array(this.hostCount).fill(false); + return { + pickNext: () => { + const i = this.best(local); + if (i === null) return null; + local[i] = true; + return i; + }, + }; + } +} +``` + +- [ ] **Step 4: Run test** → PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/hostTracker.ts test/qwp/hostTracker.test.ts +git commit -m "feat(qwp): add state-ranked host tracker with private cursors" +``` + +--- + +### Task 8: Reconnect, rotation, and dictionary catch-up + +**Files:** +- Modify: `src/qwp/transport.ts` +- Test: `test/qwp/transport.reconnect.test.ts` + +Spec 6.5.1, 7.5. A `421` role reject retries **indefinitely**; `401`/`403` is terminal. After every reconnect, a **dictionary catch-up frame** re-registers from id 0 before any data frame — the server's dictionary is connection-scoped and empty. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/transport.reconnect.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { MockQwpServer } from "./mockServer"; +import { QwpTransport } from "../../src/qwp/transport"; +import { SenderOptions } from "../../src/options"; + +let mocks: MockQwpServer[] = []; +afterEach(async () => { + for (const m of mocks) await m.stop(); + mocks = []; +}); + +describe("reconnect and rotation", () => { + it("rotates to the second endpoint when the first refuses the upgrade with 421", async () => { + const bad = new MockQwpServer(); + const good = new MockQwpServer(); + mocks.push(bad, good); + const badPort = await bad.start({ + upgradeStatus: 421, + upgradeHeaders: "X-QuestDB-Role: replica\r\n", + }); + const goodPort = await good.start(); + + const t = new QwpTransport( + new SenderOptions(`ws::addr=127.0.0.1:${badPort},127.0.0.1:${goodPort};`), + ); + await t.connect(); + expect(t.connectedEndpoint!.port).toBe(goodPort); + await t.close(); + }); + + it("fails terminally on 401 without rotating", async () => { + const a = new MockQwpServer(); + mocks.push(a); + const port = await a.start({ upgradeStatus: 401 }); + const t = new QwpTransport(new SenderOptions(`ws::addr=127.0.0.1:${port};`)); + await expect(t.connect()).rejects.toThrow(/authentication/i); + }); + + it("sends a dictionary catch-up frame before data after reconnect", async () => { + const s = new MockQwpServer(); + mocks.push(s); + const port = await s.start(); + const t = new QwpTransport(new SenderOptions(`ws::addr=127.0.0.1:${port};`)); + await t.connect(); + t.registerSymbolForTest("alpha"); + await t.reconnectForTest(); + // First frame after reconnect must carry the dictionary from id 0. + const first = s.frames[0]; + expect(first.subarray(0, 4).toString("ascii")).toBe("QWP1"); + await t.close(); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, `connectedEndpoint` undefined. + +- [ ] **Step 3: Implement** — in `src/qwp/transport.ts`: + +```ts + private endpoints: Endpoint[] = []; + private tracker!: HostTracker; + private current?: Endpoint; + private readonly dict = new SymbolDict(); + private confirmedMaxId = -1; + + get connectedEndpoint(): Endpoint | undefined { + return this.current; + } + + async connect(): Promise { + this.endpoints = parseAddrList(this.options.addr!, 9000); + this.tracker = new HostTracker(this.endpoints.length); + return this.connectLoop(); + } + + private async connectLoop(): Promise { + for (;;) { + const idx = this.tracker.pickNext(); + if (idx === null) { + this.tracker.beginRound(); + // A 421 round means no primary is reachable yet; retry indefinitely + // rather than giving up (spec 6.5.1). + await new Promise((r) => setTimeout(r, this.backoffMillis())); + continue; + } + const ep = this.endpoints[idx]; + try { + this.ws = await QwpWebSocket.connect({ ...this.wsOptions(ep) }); + this.tracker.record(idx, HostState.HEALTHY); + this.current = ep; + this.acks.onConnected(this.acks.acked + 1); + await this.sendDictCatchUp(); + return true; + } catch (e) { + if (e instanceof QwpUpgradeError) { + if (e.kind === "auth") throw e; // terminal, never rotate + this.tracker.record( + idx, + e.kind === "role-reject" ? HostState.TOPOLOGY_REJECT : HostState.TRANSPORT_ERROR, + ); + } else { + this.tracker.record(idx, HostState.TRANSPORT_ERROR); + } + } + } + } + + /** + * The server's dictionary is connection-scoped and empty after a reconnect, + * so re-register from id 0 before any data frame or every delta frame earns + * DICTIONARY_GAP (spec 7.5). + */ + private async sendDictCatchUp(): Promise { + if (this.dict.size() === 0) return; + const cap = this.ws?.maxBatchSize ?? UNCAPPED_CATCHUP_PACKING_LIMIT; + const frame = encodeFrame([], { gorilla: false, dict: this.dict, confirmedMaxId: -1 }); + if (frame.length > cap) { + throw new Error(`dictionary catch-up exceeds the batch cap [size=${frame.length}, cap=${cap}]`); + } + await this.ws!.sendBinary(frame); + this.confirmedMaxId = this.dict.size() - 1; + } +``` + +Add `export const UNCAPPED_CATCHUP_PACKING_LIMIT = 64 * 1024;` to `constants.ts` — "not advertised" is **not** "unbounded" (spec 7.5). Add the two test hooks `registerSymbolForTest` and `reconnectForTest`. + +- [ ] **Step 4: Run test** → PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/transport.ts src/qwp/protocol/constants.ts test/qwp/transport.reconnect.test.ts +git commit -m "feat(qwp): add reconnect, endpoint rotation and dictionary catch-up" +``` + +--- + +### Task 9: Notification dispatchers and connect-mode derivation + +**Files:** +- Create: `src/qwp/dispatcher.ts` +- Modify: `src/qwp/transport.ts`, `src/sender.ts` +- Test: `test/qwp/dispatcher.test.ts` + +**Spec 4.2 — the inbox drops the OLDEST**, not the newest. Watermarks are monotonic, so the newest entry is the most informative; drop-newest retains stale state under load. **Spec 4.3** — connect mode is *derived*: any `reconnect_*` key set implies eager connect. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/dispatcher.test.ts +import { describe, it, expect } from "vitest"; +import { Dispatcher } from "../../src/qwp/dispatcher"; +import { SenderOptions } from "../../src/options"; +import { deriveConnectMode, ConnectMode } from "../../src/qwp/transport"; + +describe("Dispatcher", () => { + it("drops the OLDEST entry when full and counts the drop", async () => { + const seen: number[] = []; + const d = new Dispatcher(2, (v) => seen.push(v)); + d.offer(1); + d.offer(2); + d.offer(3); // evicts 1 + await new Promise((r) => setImmediate(r)); + expect(seen).toEqual([2, 3]); + expect(d.dropped).toBe(1); + }); + + it("never invokes the handler synchronously", () => { + let called = false; + const d = new Dispatcher(4, () => (called = true)); + d.offer(1); + expect(called).toBe(false); + }); + + it("survives a throwing handler", async () => { + const d = new Dispatcher(4, () => { + throw new Error("bad handler"); + }); + d.offer(1); + await new Promise((r) => setImmediate(r)); + expect(d.dropped).toBe(0); + }); +}); + +describe("connect mode derivation", () => { + it("is OFF when no reconnect_* key is set", () => { + expect(deriveConnectMode(new SenderOptions("ws::addr=h:9000;"))).toBe(ConnectMode.OFF); + }); + + it("is SYNC when any reconnect_* key is supplied", () => { + expect( + deriveConnectMode(new SenderOptions("ws::addr=h:9000;reconnect_max_backoff_millis=1000;")), + ).toBe(ConnectMode.SYNC); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, modules not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/dispatcher.ts + +/** + * Bounded inbox that drops the OLDEST entry when full (spec 4.2). Watermarks + * are monotonic, so the newest entry is always the most informative and + * dropping the head compresses information rather than losing it. + * The handler is never invoked on the caller's stack. + */ +export class Dispatcher { + private readonly queue: T[] = []; + private scheduled = false; + dropped = 0; + + constructor( + private readonly capacity: number, + private readonly handler: (item: T) => void, + ) { + if (capacity < 1) throw new Error("capacity must be >= 1"); + } + + offer(item: T): void { + if (this.queue.length >= this.capacity) { + this.queue.shift(); + this.dropped++; + } + this.queue.push(item); + if (!this.scheduled) { + this.scheduled = true; + setImmediate(() => this.drain()); + } + } + + private drain(): void { + this.scheduled = false; + while (this.queue.length > 0) { + const item = this.queue.shift()!; + try { + this.handler(item); + } catch { + /* a handler must never break the sender */ + } + } + } +} +``` + +In `src/qwp/transport.ts`: + +```ts +export enum ConnectMode { + OFF = "OFF", + SYNC = "SYNC", +} + +/** + * The default is DERIVED: setting any reconnect_* key implicitly upgrades + * construction from non-connecting to connecting-with-retry, because those + * knobs read as a general retry budget while the underlying path governs only + * reconnects from an established connection (spec 4.3). + */ +export function deriveConnectMode(o: SenderOptions): ConnectMode { + const anyReconnect = + o.reconnect_max_duration_millis !== undefined || + o.reconnect_initial_backoff_millis !== undefined || + o.reconnect_max_backoff_millis !== undefined; + return anyReconnect ? ConnectMode.SYNC : ConnectMode.OFF; +} +``` + +Replace the raw `errorHandler` with `new Dispatcher(o.error_inbox_capacity ?? 256, h)` and add a connection-event dispatcher at capacity **64** (spec 9.1 — the two differ). + +- [ ] **Step 4: Run test** → PASS, 5 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/dispatcher.ts src/qwp/transport.ts src/sender.ts test/qwp/dispatcher.test.ts +git commit -m "feat(qwp): add drop-oldest dispatchers and derived connect mode" +``` + +--- + +## Self-Review + +**1. Spec coverage.** Response decode (Task 1 — 6.6). Categories/policies (Task 2 — 7.1–7.3). ACK→FSN (Task 3 — 6.6.1). NACK handling + in-flight loss (Task 4). Poison detector, both conditions (Task 5 — 7.4). `addr` grammar (Task 6 — 1.2). Host tracker, state-only, private cursors (Task 7 — 1.2). Reconnect, rotation, catch-up (Task 8 — 6.5.1, 7.5). Dispatchers + connect mode (Task 9 — 4.2, 4.3, 9.1). + +**Explicitly deferred to Plan 4, with reasons stated in the sequencing note:** replay from `ackedFsn + 1` (needs the ring), the `.symbol-dict` delta→full-dict fallback (needs the file), `DATA_LOSS`/`ABANDONED` from a quarantined slot (Task 4 emits `DATA_LOSS` only for in-flight loss). + +**2. Placeholder scan.** None. + +**3. Type consistency.** `STATUS` (Task 1) is used by `classify` (Task 2) and the mock server (Task 4). `Category`/`Policy`/`SenderError` (Task 2) are used in Tasks 4 and 9. `AckTracker.onConnected/onFrameSent/onAck` (Task 3) is called in Tasks 4 and 8. `Endpoint`/`parseAddrList` (Task 6) feed `HostTracker` (Task 7) and the connect loop (Task 8). `MockQwpServer` (Task 4) is reused in Task 8. diff --git a/docs/superpowers/plans/2026-08-07-qwp-plan-4-store-and-forward.md b/docs/superpowers/plans/2026-08-07-qwp-plan-4-store-and-forward.md new file mode 100644 index 0000000..7655114 --- /dev/null +++ b/docs/superpowers/plans/2026-08-07-qwp-plan-4-store-and-forward.md @@ -0,0 +1,1196 @@ +# QWP Plan 4 — Store-and-Forward and Release (spec PRs 12–16) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `flush()`'s publish semantics honest — published frames survive a disconnect and a process crash, replay after reconnect, and a crashed process's leftover data gets drained. Then ship 4.3.0. + +**Architecture:** A byte-capped ring of segments keyed by FSN, in memory or on disk depending on whether `sf_dir` is set. Segments carry `baseSeq` in their header, so **FSNs persist across restarts**. Two crash-safe boundary records (manifest, ack watermark) and a load-bearing persisted symbol dictionary sit beside them. Slot directories are locked, and orphaned slots are drained by background tasks with their own connections. + +**Tech Stack:** TypeScript, Node ≥ 20, `node:fs/promises`, `node:crypto`. vitest + testcontainers + child-process crash tests. + +**Prerequisites:** Plans 1–3 merged. Consumes: `SymbolDict`, `AckTracker`, `HostTracker` (and its `newCursor()`), `encodeFrame`, `QwpTransport`, `Dispatcher`, `SenderError`. + +**Source of truth:** `docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md`. + +## Global Constraints + +- **No new runtime dependencies.** In particular **no CRC library** — `zlib.crc32` is ISO-HDLC and will not interoperate; CRC32C (Castagnoli) is implemented in Task 1. +- **Node 20 floor.** +- **All integers little-endian.** +- **Options stay `undefined` until set** (spec 9.1.2). +- **Mode selection is implicit:** `sf_dir` present ⇒ disk mode; absent ⇒ memory mode. There is no `store_and_forward` key (spec 9.2). +- Existing tests stay green: `npx vitest run && npx tsc --noEmit && npx eslint src/**`. + +## File Structure + +| File | Responsibility | +|---|---| +| `src/qwp/sf/crc32c.ts` | CRC32C (Castagnoli) | +| `src/qwp/sf/segment.ts` | `SF01` segment: append, scan-recover, torn-tail detection | +| `src/qwp/sf/ring.ts` | FSN-keyed segment chain, ACK-driven trim, publish barrier | +| `src/qwp/sf/boundary.ts` | Alternating-generation record (manifest + ack watermark) | +| `src/qwp/sf/symbolDictFile.ts` | `SYD1` persisted dictionary | +| `src/qwp/sf/slotLock.ts` | Slot lock + logical lock | +| `src/qwp/sf/orphans.ts` | Orphan scan + background drainers | +| `src/qwp/sf/engine.ts` | Ties ring + manager + watermark together | +| `src/qwp/transport.ts` | **modify** — publish to the engine, replay on reconnect | +| `README.md` | **modify** — support matrix and the caveats in Task 10 | + +--- + +### Task 1: CRC32C + +**Files:** +- Create: `src/qwp/sf/crc32c.ts` +- Test: `test/qwp/sf/crc32c.test.ts` + +**Interfaces:** +- Produces: `crc32c(buf: Buffer, seed?: number): number`. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/crc32c.test.ts +import { describe, it, expect } from "vitest"; +import { crc32c } from "../../../src/qwp/sf/crc32c"; + +describe("crc32c (Castagnoli)", () => { + it("matches the published check value for '123456789'", () => { + expect(crc32c(Buffer.from("123456789", "ascii")) >>> 0).toBe(0xe3069283); + }); + + it("returns 0 for an empty buffer", () => { + expect(crc32c(Buffer.alloc(0)) >>> 0).toBe(0); + }); + + it("is order-sensitive", () => { + expect(crc32c(Buffer.from("ab"))).not.toBe(crc32c(Buffer.from("ba"))); + }); +}); +``` + +- [ ] **Step 2: Run test** → `npx vitest run test/qwp/sf/crc32c.test.ts` → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/crc32c.ts +import { Buffer } from "node:buffer"; + +// Castagnoli polynomial, reversed: 0x82F63B78. NOT the zlib/ISO-HDLC one. +const TABLE = (() => { + const t = new Int32Array(256); + for (let i = 0; i < 256; i++) { + let c = i; + for (let k = 0; k < 8; k++) c = c & 1 ? 0x82f63b78 ^ (c >>> 1) : c >>> 1; + t[i] = c; + } + return t; +})(); + +export function crc32c(buf: Buffer, seed = 0): number { + let c = ~seed; + for (let i = 0; i < buf.length; i++) { + c = TABLE[(c ^ buf[i]) & 0xff] ^ (c >>> 8); + } + return (~c) >>> 0; +} +``` + +- [ ] **Step 4: Run test** → PASS, 3 tests. The `123456789` check value is the published CRC-32C constant, so this validates against the standard rather than against our own output. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/crc32c.ts test/qwp/sf/crc32c.test.ts +git commit -m "feat(qwp): add CRC32C (Castagnoli) implementation" +``` + +--- + +### Task 2: Segment format, append and torn-tail recovery + +**Files:** +- Create: `src/qwp/sf/segment.ts` +- Test: `test/qwp/sf/segment.test.ts` + +**Spec 8.1.5.** 24-byte `SF01` header, then frames of `u32 crc32c | u32 payloadLen | payload`, with the **CRC covering `payloadLen` and the payload together**. Recovery stops at the first bad CRC or a length that overruns the file. **The residue must not be zeroed during the scan** — it can hold valid-CRC frames that are the only surviving copy. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/segment.test.ts +import { describe, it, expect } from "vitest"; +import { buildSegment, scanSegment, SEGMENT_HEADER_SIZE } from "../../../src/qwp/sf/segment"; + +function seg(baseSeq: number, frames: Buffer[]): Buffer { + return buildSegment(baseSeq, frames, 4096); +} + +describe("segment", () => { + it("writes an SF01 header carrying baseSeq", () => { + const b = seg(42, [Buffer.from("aa")]); + expect(b.subarray(0, 4).toString("ascii")).toBe("SF01"); + expect(b.readUInt8(4)).toBe(1); + expect(Number(b.readBigUInt64LE(8))).toBe(42); + }); + + it("scans back the frames it wrote", () => { + const r = scanSegment(seg(0, [Buffer.from("aa"), Buffer.from("bbb")])); + expect(r.frames.map((f) => f.toString())).toEqual(["aa", "bbb"]); + expect(r.tornTailBytes).toBe(0); + }); + + it("stops at a bad CRC and reports a torn tail", () => { + const b = seg(0, [Buffer.from("aa"), Buffer.from("bbb")]); + b[SEGMENT_HEADER_SIZE + 8 + 2 + 0] ^= 0xff; // corrupt the second frame's payload + const r = scanSegment(b); + expect(r.frames.length).toBe(1); + expect(r.tornTailBytes).toBeGreaterThan(0); + }); + + it("distinguishes a clean partial fill from a torn tail", () => { + const b = seg(0, [Buffer.from("aa")]); + // trailing bytes are already zero -> clean fill, not a tear + expect(scanSegment(b).tornTailBytes).toBe(0); + }); + + it("stops when a declared length overruns the file", () => { + const b = seg(0, [Buffer.from("aa")]); + b.writeUInt32LE(9_000_000, SEGMENT_HEADER_SIZE + 4); + expect(scanSegment(b).frames.length).toBe(0); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/segment.ts +import { Buffer } from "node:buffer"; +import { crc32c } from "./crc32c"; + +export const SEGMENT_MAGIC = Buffer.from("SF01", "ascii"); +export const SEGMENT_HEADER_SIZE = 24; +export const FRAME_HEADER_SIZE = 8; + +export interface ScanResult { + baseSeq: number; + frames: Buffer[]; + /** Bytes of non-zero residue after the last valid frame. 0 = clean fill. */ + tornTailBytes: number; + /** Offset where the next append must start. */ + appendOffset: number; +} + +export function buildSegment(baseSeq: number, frames: Buffer[], capacity: number): Buffer { + const buf = Buffer.alloc(capacity); + SEGMENT_MAGIC.copy(buf, 0); + buf.writeUInt8(1, 4); // version + buf.writeUInt8(0, 5); // flags + buf.writeUInt16LE(0, 6); // reserved + buf.writeBigUInt64LE(BigInt(baseSeq), 8); + buf.writeBigUInt64LE(0n, 16); // createdMicros; stamped by the caller if needed + let o = SEGMENT_HEADER_SIZE; + for (const f of frames) o = appendFrame(buf, o, f); + return buf; +} + +/** Returns the new offset, or -1 when the frame does not fit. */ +export function appendFrame(buf: Buffer, offset: number, payload: Buffer): number { + const need = FRAME_HEADER_SIZE + payload.length; + if (offset + need > buf.length) return -1; + // CRC covers (payloadLen, payload) together -- not the payload alone. + const lenAndPayload = Buffer.allocUnsafe(4 + payload.length); + lenAndPayload.writeUInt32LE(payload.length, 0); + payload.copy(lenAndPayload, 4); + buf.writeUInt32LE(crc32c(lenAndPayload), offset); + buf.writeUInt32LE(payload.length, offset + 4); + payload.copy(buf, offset + 8); + return offset + need; +} + +export function scanSegment(buf: Buffer): ScanResult { + if (buf.length < SEGMENT_HEADER_SIZE || !buf.subarray(0, 4).equals(SEGMENT_MAGIC)) { + throw new Error("segment: bad magic"); + } + if (buf.readUInt8(4) !== 1) throw new Error("segment: unsupported version"); + const baseSeq = Number(buf.readBigUInt64LE(8)); + + const frames: Buffer[] = []; + let o = SEGMENT_HEADER_SIZE; + for (;;) { + if (o + FRAME_HEADER_SIZE > buf.length) break; + const crc = buf.readUInt32LE(o); + const len = buf.readUInt32LE(o + 4); + if (len === 0 && crc === 0) break; // unwritten space + if (o + FRAME_HEADER_SIZE + len > buf.length) break; // declared length overruns + const lenAndPayload = buf.subarray(o + 4, o + FRAME_HEADER_SIZE + len); + if (crc32c(lenAndPayload) !== crc) break; // first bad CRC ends the chain + frames.push(Buffer.from(buf.subarray(o + FRAME_HEADER_SIZE, o + FRAME_HEADER_SIZE + len))); + o += FRAME_HEADER_SIZE + len; + } + + // Non-zero residue means a write was attempted and failed. Do NOT zero it + // here: after a mid-file tear it can hold valid-CRC frames that are the only + // surviving copy of real payloads (spec 8.1.5). + let torn = 0; + for (let i = o; i < buf.length; i++) if (buf[i] !== 0) torn++; + + return { baseSeq, frames, tornTailBytes: torn, appendOffset: o }; +} +``` + +- [ ] **Step 4: Run test** → PASS, 5 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/segment.ts test/qwp/sf/segment.test.ts +git commit -m "feat(qwp): add SF01 segment format with torn-tail recovery" +``` + +--- + +### Task 3: Segment ring and FSN model + +**Files:** +- Create: `src/qwp/sf/ring.ts` +- Test: `test/qwp/sf/ring.test.ts` + +**Spec 8.1.1.** FSNs derive from `baseSeq + frameCount`, so a recovered ring **continues** the previous numbering. Two sentinels with opposite handling: no-spare means wait, payload-too-large means fail now. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/ring.test.ts +import { describe, it, expect } from "vitest"; +import { SegmentRing, BACKPRESSURE_NO_SPARE, PAYLOAD_TOO_LARGE } from "../../../src/qwp/sf/ring"; + +describe("SegmentRing", () => { + it("assigns FSNs from 0 on a fresh ring", () => { + const r = new SegmentRing({ segmentBytes: 4096, maxTotalBytes: 1 << 20 }); + expect(r.publishedFsn).toBe(-1); + expect(r.append(Buffer.from("a"))).toBe(0); + expect(r.append(Buffer.from("b"))).toBe(1); + expect(r.publishedFsn).toBe(1); + }); + + it("continues numbering when recovered from existing segments", () => { + const r = SegmentRing.recovered([{ baseSeq: 10, frames: [Buffer.from("x"), Buffer.from("y")] }], { + segmentBytes: 4096, + maxTotalBytes: 1 << 20, + }); + expect(r.publishedFsn).toBe(11); + expect(r.append(Buffer.from("z"))).toBe(12); + }); + + it("returns PAYLOAD_TOO_LARGE for a frame that cannot fit a fresh segment", () => { + const r = new SegmentRing({ segmentBytes: 64, maxTotalBytes: 1 << 20 }); + expect(r.append(Buffer.alloc(1000))).toBe(PAYLOAD_TOO_LARGE); + }); + + it("trims acked segments and frees space", () => { + const r = new SegmentRing({ segmentBytes: 64, maxTotalBytes: 256 }); + const fsns = [0, 1, 2].map(() => r.append(Buffer.alloc(20))); + r.acknowledge(fsns[2]); + expect(r.ackedFsn).toBe(fsns[2]); + expect(r.totalBytes).toBeLessThan(256); + }); + + it("returns the frames to replay from ackedFsn + 1", () => { + const r = new SegmentRing({ segmentBytes: 4096, maxTotalBytes: 1 << 20 }); + r.append(Buffer.from("a")); + r.append(Buffer.from("b")); + r.append(Buffer.from("c")); + r.acknowledge(0); + expect(r.framesFrom(r.ackedFsn + 1).map((f) => f.toString())).toEqual(["b", "c"]); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/ring.ts +import { Buffer } from "node:buffer"; + +export const BACKPRESSURE_NO_SPARE = -1; +export const PAYLOAD_TOO_LARGE = -2; + +interface Seg { + baseSeq: number; + frames: Buffer[]; + bytes: number; +} + +export interface RingOptions { + segmentBytes: number; + maxTotalBytes: number; +} + +export class SegmentRing { + private segs: Seg[] = []; + private nextSeq = 0; + private acked = -1; + + constructor(private readonly opts: RingOptions) { + this.segs.push({ baseSeq: 0, frames: [], bytes: 0 }); + } + + /** FSNs derive from the chain, so a recovered ring continues numbering. */ + static recovered( + chain: { baseSeq: number; frames: Buffer[] }[], + opts: RingOptions, + ): SegmentRing { + const r = new SegmentRing(opts); + r.segs = chain.map((c) => ({ + baseSeq: c.baseSeq, + frames: c.frames, + bytes: c.frames.reduce((a, f) => a + f.length, 0), + })); + r.segs.sort((a, b) => a.baseSeq - b.baseSeq); + for (let i = 1; i < r.segs.length; i++) { + const prev = r.segs[i - 1]; + if (prev.baseSeq + prev.frames.length !== r.segs[i].baseSeq) { + throw new Error("segment chain is not contiguous"); + } + } + for (const s of r.segs) { + if (s.baseSeq < 0) throw new Error("segment with negative baseSeq must be quarantined"); + } + const last = r.segs[r.segs.length - 1]; + r.nextSeq = last.baseSeq + last.frames.length; + return r; + } + + get publishedFsn(): number { + return this.nextSeq - 1; + } + + get ackedFsn(): number { + return this.acked; + } + + get totalBytes(): number { + return this.segs.reduce((a, s) => a + s.bytes, 0); + } + + /** Returns the assigned FSN, or a negative sentinel. */ + append(frame: Buffer): number { + if (frame.length > this.opts.segmentBytes) return PAYLOAD_TOO_LARGE; + const active = this.segs[this.segs.length - 1]; + if (active.bytes + frame.length > this.opts.segmentBytes) { + if (this.totalBytes + frame.length > this.livenessFloorAdjustedCap()) { + return BACKPRESSURE_NO_SPARE; + } + this.segs.push({ baseSeq: this.nextSeq, frames: [], bytes: 0 }); + } + const seg = this.segs[this.segs.length - 1]; + seg.frames.push(frame); + seg.bytes += frame.length; + return this.nextSeq++; + } + + /** + * Never refuse below the minimum working set. Segment bytes are reclaimable + * by ACK-driven trim, but side files are lifetime-monotonic, so refusing on + * the raw total can wedge the producer permanently (spec 8.1.3). + */ + private livenessFloorAdjustedCap(): number { + return Math.max(this.opts.maxTotalBytes, 2 * this.opts.segmentBytes); + } + + acknowledge(fsn: number): void { + if (fsn > this.acked) this.acked = fsn; + while (this.segs.length > 1) { + const head = this.segs[0]; + if (head.baseSeq + head.frames.length - 1 > this.acked) break; + this.segs.shift(); + } + } + + framesFrom(fsn: number): Buffer[] { + const out: Buffer[] = []; + for (const s of this.segs) { + s.frames.forEach((f, i) => { + if (s.baseSeq + i >= fsn) out.push(f); + }); + } + return out; + } +} +``` + +- [ ] **Step 4: Run test** → PASS, 5 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/ring.ts test/qwp/sf/ring.test.ts +git commit -m "feat(qwp): add FSN-keyed segment ring with ACK-driven trim" +``` + +--- + +### Task 4: Replay on reconnect + +**Files:** +- Modify: `src/qwp/transport.ts` +- Test: `test/qwp/sf/replay.test.ts` + +The gap Plan 3 left open. `flush()` publishes into the ring; the send loop drains it; a reconnect replays from `ackedFsn + 1` **after** the dictionary catch-up. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/replay.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { MockQwpServer } from "../mockServer"; +import { QwpTransport } from "../../../src/qwp/transport"; +import { SenderOptions } from "../../../src/options"; + +let mock: MockQwpServer | undefined; +afterEach(async () => await mock?.stop()); + +describe("replay", () => { + it("resends unacked frames after a reconnect", async () => { + mock = new MockQwpServer(); + // Drop the connection after the first frame, never ACKing it. + const port = await mock.start({ dropAfter: 1 }); + const t = new QwpTransport(new SenderOptions(`ws::addr=127.0.0.1:${port};`)); + await t.connect(); + await t.sendFrames([Buffer.from("QWP1frame-one")]); + await new Promise((r) => setTimeout(r, 400)); + // The same payload must appear at least twice: original plus replay. + const matching = mock.frames.filter((f) => f.toString().includes("frame-one")); + expect(matching.length).toBeGreaterThanOrEqual(2); + await t.close(); + }); + + it("does not replay frames the server already acked", async () => { + mock = new MockQwpServer(); + const port = await mock.start(); + const t = new QwpTransport(new SenderOptions(`ws::addr=127.0.0.1:${port};`)); + await t.connect(); + await t.sendFrames([Buffer.from("QWP1acked-frame")]); + await new Promise((r) => setTimeout(r, 150)); + await t.reconnectForTest(); + await new Promise((r) => setTimeout(r, 150)); + const matching = mock.frames.filter((f) => f.toString().includes("acked-frame")); + expect(matching.length).toBe(1); + await t.close(); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL — no ring wired in, so nothing replays. + +- [ ] **Step 3: Implement** — in `src/qwp/transport.ts`: + +```ts + private ring = new SegmentRing({ + segmentBytes: 4 * 1024 * 1024, + maxTotalBytes: 128 * 1024 * 1024, // memory-mode default (spec 9.1) + }); + + async sendFrames(frames: Buffer[]): Promise { + for (const f of frames) { + const fsn = this.ring.append(f); + if (fsn === PAYLOAD_TOO_LARGE) { + throw new Error(`frame does not fit a fresh segment [size=${f.length}]`); + } + if (fsn === BACKPRESSURE_NO_SPARE) { + await this.awaitSpace(); + this.ring.append(f); + } + } + await this.drain(); + return true; + } + + /** Sends everything published beyond what has been sent on this connection. */ + private async drain(): Promise { + if (!this.ws) return; + const pending = this.ring.framesFrom(this.sentUpTo + 1); + for (const f of pending) { + await this.ws.sendBinary(f); + this.acks.onFrameSent(); + this.sentUpTo++; + } + } + + private async onReconnected(): Promise { + await this.sendDictCatchUp(); // dictionary first (spec 7.5) + this.sentUpTo = this.ring.ackedFsn; // replay from ackedFsn + 1 + this.acks.onConnected(this.ring.ackedFsn + 1); + await this.drain(); + } +``` + +Route `onResponse`'s OK path through `this.ring.acknowledge(fsn)` using the FSN returned by `AckTracker.onAck`, and drop the Plan 3 `DATA_LOSS`-on-disconnect emission — with retention in place, a disconnect no longer loses in-flight frames. + +- [ ] **Step 4: Run test** → PASS, 2 tests. Also re-run `test/qwp/transport.acks.test.ts` and **update** its "reports in-flight loss" case: that behaviour is now intentionally gone, replaced by replay. Rewrite it to assert the frame is *replayed* rather than reported lost. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/transport.ts test/qwp/sf/replay.test.ts test/qwp/transport.acks.test.ts +git commit -m "feat(qwp): replay unacked frames from the ring after reconnect" +``` + +--- + +### Task 5: Crash-safe boundary record + +**Files:** +- Create: `src/qwp/sf/boundary.ts` +- Test: `test/qwp/sf/boundary.test.ts` + +**Spec 8.2.** Two independently CRC-protected 64-byte records at offsets **0 and 4096**, alternating on update, CRC written last. Recovery picks the valid record with the greatest generation. The 4 KiB separation stops one sector tear damaging both. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/boundary.test.ts +import { describe, it, expect } from "vitest"; +import { writeBoundary, readBoundary, BOUNDARY_FILE_SIZE } from "../../../src/qwp/sf/boundary"; + +describe("boundary record", () => { + it("alternates slots and picks the greatest valid generation", () => { + const buf = Buffer.alloc(BOUNDARY_FILE_SIZE); + writeBoundary(buf, 1, 100n); + writeBoundary(buf, 2, 200n); + expect(readBoundary(buf)).toEqual({ generation: 2, value: 200n }); + }); + + it("writes the two records 4096 bytes apart", () => { + const buf = Buffer.alloc(BOUNDARY_FILE_SIZE); + writeBoundary(buf, 1, 100n); + writeBoundary(buf, 2, 200n); + expect(buf.readUInt32LE(0)).not.toBe(0); + expect(buf.readUInt32LE(4096)).not.toBe(0); + }); + + it("falls back to the older record when the newer one is torn", () => { + const buf = Buffer.alloc(BOUNDARY_FILE_SIZE); + writeBoundary(buf, 1, 100n); + writeBoundary(buf, 2, 200n); + // Corrupt whichever slot holds generation 2. + const slot = buf.readBigUInt64LE(8) === 2n ? 0 : 4096; + buf[slot + 20] ^= 0xff; + expect(readBoundary(buf)).toEqual({ generation: 1, value: 100n }); + }); + + it("returns null when neither record validates", () => { + const buf = Buffer.alloc(BOUNDARY_FILE_SIZE); + expect(readBoundary(buf)).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/boundary.ts +import { Buffer } from "node:buffer"; +import { crc32c } from "./crc32c"; + +export const BOUNDARY_FILE_SIZE = 8192; +const RECORD_SIZE = 64; +const SLOT_STRIDE = 4096; +const CRC_OFFSET = 60; +const MAGIC = 0x314b5741; // 'AKW1' + +export interface Boundary { + generation: number; + value: bigint; +} + +/** Alternates slots by generation parity; the CRC is written last. */ +export function writeBoundary(buf: Buffer, generation: number, value: bigint): void { + const slot = (generation % 2) * SLOT_STRIDE; + buf.fill(0, slot, slot + RECORD_SIZE); + buf.writeUInt32LE(MAGIC, slot); + buf.writeUInt32LE(1, slot + 4); + buf.writeBigUInt64LE(BigInt(generation), slot + 8); + buf.writeBigInt64LE(value, slot + 16); + const crc = crc32c(buf.subarray(slot, slot + CRC_OFFSET)); + buf.writeUInt32LE(crc, slot + CRC_OFFSET); +} + +export function readBoundary(buf: Buffer): Boundary | null { + let best: Boundary | null = null; + for (const slot of [0, SLOT_STRIDE]) { + if (slot + RECORD_SIZE > buf.length) continue; + if (buf.readUInt32LE(slot) !== MAGIC) continue; + const stored = buf.readUInt32LE(slot + CRC_OFFSET); + if (crc32c(buf.subarray(slot, slot + CRC_OFFSET)) !== stored) continue; + const generation = Number(buf.readBigUInt64LE(slot + 8)); + const value = buf.readBigInt64LE(slot + 16); + if (!best || generation > best.generation) best = { generation, value }; + } + return best; +} +``` + +- [ ] **Step 4: Run test** → PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/boundary.ts test/qwp/sf/boundary.test.ts +git commit -m "feat(qwp): add alternating-generation crash-safe boundary record" +``` + +--- + +### Task 6: Persisted symbol dictionary + +**Files:** +- Create: `src/qwp/sf/symbolDictFile.ts` +- Test: `test/qwp/sf/symbolDictFile.test.ts` + +**Spec 8.1.6 — load-bearing, not an optimisation.** `SYD1` header then chunks of `[entryCount varint][entryBytes varint][entries][crc32c u32]`. **One chunk = one frame's new symbols.** Ids are implicit and positional, so **recovery must not de-duplicate**. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/symbolDictFile.test.ts +import { describe, it, expect } from "vitest"; +import { encodeChunk, decodeDictFile, DICT_HEADER } from "../../../src/qwp/sf/symbolDictFile"; +import { SymbolDict } from "../../../src/qwp/protocol/symbolDict"; + +describe("persisted symbol dictionary", () => { + it("round-trips chunks in order", () => { + const file = Buffer.concat([DICT_HEADER, encodeChunk(["a", "b"]), encodeChunk(["c"])]); + expect(decodeDictFile(file)).toEqual(["a", "b", "c"]); + }); + + it("stops at the first bad chunk CRC, keeping the prefix", () => { + const file = Buffer.concat([DICT_HEADER, encodeChunk(["a"]), encodeChunk(["b"])]); + file[file.length - 1] ^= 0xff; + expect(decodeDictFile(file)).toEqual(["a"]); + }); + + it("recovery preserves positional ids and does NOT de-duplicate", () => { + const file = Buffer.concat([DICT_HEADER, encodeChunk(["x", "x"])]); + const entries = decodeDictFile(file); + const dict = new SymbolDict(); + for (const e of entries) dict.addRecovered(e); + expect(dict.size()).toBe(2); // collapsing would renumber every later symbol + }); + + it("rejects a bad magic", () => { + expect(() => decodeDictFile(Buffer.from("NOPE0000"))).toThrow(/magic/i); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/symbolDictFile.ts +import { Buffer } from "node:buffer"; +import { crc32c } from "./crc32c"; +import { writeVarint, varintSize, readVarint } from "../protocol/varint"; + +export const DICT_HEADER = (() => { + const b = Buffer.alloc(8); + b.write("SYD1", 0, "ascii"); + b.writeUInt8(1, 4); + return b; +})(); + +/** One chunk = exactly the symbols one frame introduces (spec 8.1.6). */ +export function encodeChunk(entries: string[]): Buffer { + let entryBytes = 0; + for (const s of entries) { + const n = Buffer.byteLength(s, "utf8"); + entryBytes += varintSize(n) + n; + } + const head = Buffer.alloc(varintSize(entries.length) + varintSize(entryBytes)); + let ho = writeVarint(head, 0, entries.length); + ho = writeVarint(head, ho, entryBytes); + + const body = Buffer.alloc(entryBytes); + let bo = 0; + for (const s of entries) { + const n = Buffer.byteLength(s, "utf8"); + bo = writeVarint(body, bo, n); + body.write(s, bo, "utf8"); + bo += n; + } + + // CRC covers BOTH header varints and the entry region. + const crcInput = Buffer.concat([head.subarray(0, ho), body]); + const tail = Buffer.alloc(4); + tail.writeUInt32LE(crc32c(crcInput), 0); + return Buffer.concat([head.subarray(0, ho), body, tail]); +} + +export function decodeDictFile(file: Buffer): string[] { + if (file.length < DICT_HEADER.length || file.subarray(0, 4).toString("ascii") !== "SYD1") { + throw new Error("symbol dict: bad magic"); + } + const out: string[] = []; + let o = DICT_HEADER.length; + while (o < file.length) { + const start = o; + let r; + try { + r = readVarint(file, o); + } catch { + break; + } + const count = r.value; + o = r.offset; + const r2 = readVarint(file, o); + const entryBytes = r2.value; + o = r2.offset; + if (o + entryBytes + 4 > file.length) break; + const crcInput = file.subarray(start, o + entryBytes); + if (crc32c(crcInput) !== file.readUInt32LE(o + entryBytes)) break; + + let eo = o; + for (let i = 0; i < count; i++) { + const rl = readVarint(file, eo); + eo = rl.offset; + out.push(file.subarray(eo, eo + rl.value).toString("utf8")); + eo += rl.value; + } + o += entryBytes + 4; + } + return out; +} +``` + +- [ ] **Step 4: Run test** → PASS, 4 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/symbolDictFile.ts test/qwp/sf/symbolDictFile.test.ts +git commit -m "feat(qwp): add SYD1 persisted symbol dictionary" +``` + +--- + +### Task 7: Delta → full-dict runtime fallback + +**Files:** +- Modify: `src/qwp/buffer.ts`, `src/qwp/transport.ts` +- Test: `test/qwp/sf/dictFallback.test.ts` + +**Spec 5.2.** If `.symbol-dict` becomes unwritable mid-run, degrade **permanently** to full-dict mode. Without this, every later `flush()` throws forever and a survivable condition becomes total ingestion loss. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/dictFallback.test.ts +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../../src/qwp/buffer"; +import { SymbolDict } from "../../../src/qwp/protocol/symbolDict"; +import { FLAG_DELTA_SYMBOL_DICT } from "../../../src/qwp/protocol/constants"; + +describe("delta -> full-dict fallback", () => { + it("keeps ingesting after the side file becomes unwritable", () => { + const b = new QwpBuffer(); + b.attachDict(new SymbolDict(), () => { + throw new Error("ENOSPC"); + }); + b.table("t").symbol("s", "a"); + b.at(1n, "us"); + const frames = b.sealFrames(1 << 20); // must NOT throw + expect(frames.length).toBe(1); + expect(frames[0].readUInt8(5) & FLAG_DELTA_SYMBOL_DICT).toBe(0); + }); + + it("the fallback is permanent", () => { + const b = new QwpBuffer(); + let fail = true; + b.attachDict(new SymbolDict(), () => { + if (fail) throw new Error("ENOSPC"); + }); + b.table("t").symbol("s", "a"); + b.at(1n, "us"); + b.sealFrames(1 << 20); + fail = false; // side file recovers, but we must stay in full-dict mode + b.table("t").symbol("s", "b"); + b.at(2n, "us"); + expect(b.sealFrames(1 << 20)[0].readUInt8(5) & FLAG_DELTA_SYMBOL_DICT).toBe(0); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, `attachDict` takes one argument. + +- [ ] **Step 3: Implement** — in `QwpBuffer`: + +```ts + private persist?: (entries: string[]) => void; + + attachDict(dict: SymbolDict, persist?: (entries: string[]) => void): void { + this.dict = dict; + this.persist = persist; + } + + /** + * One-way, permanent degradation. The side file can start failing appends + * while segments stay writable, because segments are pre-allocated and the + * dictionary is the one thing still growing. A fixed mode would turn that + * into total, permanent ingestion loss (spec 5.2). + */ + private disableDeltaDict(cause: unknown): void { + this.dict = undefined; + this.persist = undefined; + this.confirmedMaxId = -1; + } +``` + +In `sealFrames`, before encoding, write-ahead-persist the new symbols and fall back on failure: + +```ts + if (this.dict && this.persist) { + const fresh = this.dict.entriesFrom(this.confirmedMaxId + 1); + if (fresh.length > 0) { + try { + this.persist(fresh); + } catch (e) { + this.disableDeltaDict(e); + } + } + } +``` + +Because `symbol()` stored global ids while a dict was attached, the fallback must also re-materialise strings. Simplest correct approach: keep the string alongside the id in `col.values` as `{ id, text }` and let the encoder pick whichever the current mode needs. + +- [ ] **Step 4: Run test** → PASS, 2 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/buffer.ts src/qwp/transport.ts test/qwp/sf/dictFallback.test.ts +git commit -m "feat(qwp): degrade to full-dict mode when the symbol side file fails" +``` + +--- + +### Task 8: Disk mode — slot locks and persistence + +**Files:** +- Create: `src/qwp/sf/slotLock.ts`, `src/qwp/sf/engine.ts` +- Test: `test/qwp/sf/slotLock.test.ts`, `test/qwp/sf/engine.test.ts` + +**Spec 8.3.** A slot is `//`, `sender_id` defaults to `"default"`. Node has no `flock`, so both locks are emulated with an `O_EXCL` lockfile carrying **pid + boot id**. A second sender on the same slot must fail with a message that **names `sender_id`** as the fix. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/slotLock.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { acquireSlot, releaseSlot } from "../../../src/qwp/sf/slotLock"; + +let dir: string | undefined; +afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +describe("slot lock", () => { + it("acquires an unheld slot", async () => { + dir = mkdtempSync(join(tmpdir(), "qwp-")); + const h = await acquireSlot(dir, "default"); + expect(h).toBeTruthy(); + await releaseSlot(h); + }); + + it("refuses a second holder and names sender_id in the error", async () => { + dir = mkdtempSync(join(tmpdir(), "qwp-")); + const h = await acquireSlot(dir, "default"); + await expect(acquireSlot(dir, "default")).rejects.toThrow(/sender_id/); + await releaseSlot(h); + }); + + it("reclaims a lock from a dead pid", async () => { + dir = mkdtempSync(join(tmpdir(), "qwp-")); + const h = await acquireSlot(dir, "default"); + await releaseSlot(h); + const again = await acquireSlot(dir, "default"); + expect(again).toBeTruthy(); + await releaseSlot(again); + }); +}); +``` + +- [ ] **Step 2: Run test** → FAIL, module not found. + +- [ ] **Step 3: Implement** + +```ts +// src/qwp/sf/slotLock.ts +import { mkdir, open, readFile, unlink } from "node:fs/promises"; +import { join } from "node:path"; + +export interface SlotHandle { + slotDir: string; + lockPath: string; +} + +function bootId(): string { + // Best-effort boot identity: process start time is stable within a boot for + // a given pid, and differs across reboots for reused pids. + return String(Math.floor(Date.now() - process.uptime() * 1000)); +} + +function isAlive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch { + return false; + } +} + +/** Emulates flock; the kernel's release-on-exit is reconstructed by liveness. */ +export async function acquireSlot(sfDir: string, senderId: string): Promise { + const slotDir = join(sfDir, senderId); + await mkdir(slotDir, { recursive: true }); + const lockPath = join(slotDir, ".lock"); + + for (let attempt = 0; attempt < 2; attempt++) { + try { + const fh = await open(lockPath, "wx"); + await fh.writeFile(`${process.pid}\n${bootId()}\n`, "utf8"); + await fh.close(); + return { slotDir, lockPath }; + } catch (e: any) { + if (e.code !== "EEXIST") throw e; + const [pidStr, boot] = (await readFile(lockPath, "utf8")).split("\n"); + const pid = Number.parseInt(pidStr, 10); + const stale = boot !== bootId() || !isAlive(pid); + if (stale && attempt === 0) { + await unlink(lockPath).catch(() => undefined); + continue; + } + throw new Error( + `sf slot already in use [dir=${slotDir}, holderPid=${pid}]. ` + + `Set a distinct sender_id for each sender sharing sf_dir.`, + ); + } + } + throw new Error(`sf slot already in use [dir=${slotDir}]`); +} + +export async function releaseSlot(h: SlotHandle): Promise { + await unlink(h.lockPath).catch(() => undefined); +} +``` + +- [ ] **Step 4: Run test** → PASS, 3 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/slotLock.ts test/qwp/sf/slotLock.test.ts +git commit -m "feat(qwp): add slot locking with pid and boot-id liveness" +``` + +--- + +### Task 9: Crash-recovery tests + +**Files:** +- Create: `test/qwp/sf/crash.test.ts`, `test/qwp/sf/crashChild.ts` + +**Spec 10 tier 4.** Assert **at-least-once**, not exactly-once — replay and cap-split retry both legitimately duplicate, so a duplicate must not fail the test. + +- [ ] **Step 1: Write the failing test** + +```ts +// test/qwp/sf/crashChild.ts +import { Sender } from "../../../src"; + +async function main() { + const [addr, sfDir] = process.argv.slice(2); + const sender = await Sender.fromConfig(`ws::addr=${addr};sf_dir=${sfDir};`); + await sender.connect(); + for (let i = 0; i < 50; i++) { + await sender.table("crash_t").intColumn("i", i).at(BigInt(1_700_000_000_000_000 + i), "us"); + } + await sender.flush(); + process.stdout.write("FLUSHED\n"); + // Never exits cleanly; the parent kills us mid-flight. + await new Promise(() => undefined); +} +main(); +``` + +```ts +// test/qwp/sf/crash.test.ts +import { describe, it, expect, afterEach } from "vitest"; +import { spawn } from "node:child_process"; +import { mkdtempSync, rmSync, readdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { MockQwpServer } from "../mockServer"; + +let mock: MockQwpServer | undefined; +let dir: string | undefined; +afterEach(async () => { + await mock?.stop(); + if (dir) rmSync(dir, { recursive: true, force: true }); + dir = undefined; +}); + +describe("crash recovery", () => { + it("leaves a slot on disk when the process is killed mid-flight", async () => { + mock = new MockQwpServer(); + const port = await mock.start({ statusFor: () => 0x09 }); // never OK, so nothing trims + dir = mkdtempSync(join(tmpdir(), "qwp-sf-")); + + const child = spawn("npx", ["tsx", "test/qwp/sf/crashChild.ts", `127.0.0.1:${port}`, dir], { + cwd: process.cwd(), + }); + await new Promise((resolve) => { + child.stdout.on("data", (d) => String(d).includes("FLUSHED") && resolve()); + }); + child.kill("SIGKILL"); + await new Promise((r) => setTimeout(r, 300)); + + const slot = join(dir, "default"); + expect(readdirSync(slot).length).toBeGreaterThan(0); + }, 120_000); + + it("recovers the orphan slot and replays with no row lost", async () => { + // Assert at-least-once: every row present, duplicates allowed (spec 5.1). + expect(true).toBe(true); // placeholder replaced in Step 3 + }, 120_000); +}); +``` + +- [ ] **Step 2: Run test** → FAIL — the child cannot write a slot because disk mode is not wired. + +- [ ] **Step 3: Wire disk mode and finish the second test** + +In `src/qwp/sf/engine.ts`, open the slot when `sf_dir` is set: acquire the lock, read `.ack-watermark` via `readBoundary`, load `.symbol-dict` via `decodeDictFile` + `addRecovered`, scan `*.sfa` via `scanSegment`, and build the ring with `SegmentRing.recovered`. Replace the placeholder test with one that starts a fresh sender on the same `sf_dir`, drains against a mock that ACKs, and asserts every `i` in `0..49` appears **at least once** in the frames the mock received. + +- [ ] **Step 4: Run test** → PASS, 2 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/qwp/sf/engine.ts test/qwp/sf/crash.test.ts test/qwp/sf/crashChild.ts +git commit -m "test(qwp): add crash-recovery tests for store-and-forward" +``` + +--- + +### Task 10: Documentation and release + +**Files:** +- Modify: `README.md`, `package.json` +- Create: `examples/qwp-basic.ts` + +**These caveats must reach the README**, not only the spec — each is a case where the honest behaviour will surprise a user: + +- `flush()` resolves on **publish**, not on server ACK (spec 4.4) — unlike `http::`. +- Delivery is **at-least-once**; a cap-split retry can duplicate (spec 5.1). +- A plain OK means **server-side commit, not object-store durability**; opt into `request_durable_ack` for that (spec 4.2). +- **`sf_dir` alone is not power-loss durability** — the default `sf_durability=memory` never fsyncs (spec 8.2). +- **`drain_orphans` defaults to off**, so a crashed process's slot is never drained automatically (spec 9.1). +- **`tls_roots` cannot read a JKS keystore**; use PEM or PKCS#12 (spec 6.5.2). + +- [ ] **Step 1: Add the support-matrix row and caveats to `README.md`** + +```markdown +| Protocol | Transport | Notes | +|---|---|---| +| `http` / `https` | HTTP | ILP, request/response | +| `tcp` / `tcps` | TCP | ILP, persistent | +| `ws` / `wss` | WebSocket | **QWP** — columnar binary, store-and-forward | + +### QWP caveats + +- `flush()` resolves once rows are **published** to the send log, not when the + server acknowledges them. This differs from `http::`. +- Delivery is **at-least-once**. A retried batch can duplicate rows; use a + `DEDUP` table if you need idempotence. +- An acknowledgement means server-side **commit**, not object-store durability. + Set `request_durable_ack=on` if durability gates downstream work. +- `sf_dir` enables on-disk buffering but **not** power-loss durability on its + own — add `sf_durability=periodic`. +- `drain_orphans` is **off** by default, so a crashed process's buffered data is + not replayed automatically. +- `tls_roots` accepts PEM or PKCS#12. **JKS keystores are not supported** by + Node; convert them first. +``` + +- [ ] **Step 2: Add the example** + +```ts +// examples/qwp-basic.ts +import { Sender } from "@questdb/nodejs-client"; + +async function main() { + const sender = await Sender.fromConfig("ws::addr=localhost:9000;"); + await sender.connect(); + await sender + .table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "sell") + .floatColumn("price", 2615.54) + .floatColumn("amount", 0.00044) + .at(Date.now(), "ms"); + await sender.flush(); + await sender.close(); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); +``` + +- [ ] **Step 3: Bump the version** + +Set `"version": "4.3.0"` in `package.json`. A minor: everything added is additive. + +- [ ] **Step 4: Run the full gate** + +Run: `npx vitest run && npx tsc --noEmit && npx eslint src/** && npx bunchee` +Expected: all green, and the build emits both ESM and CJS. + +- [ ] **Step 5: Commit** + +```bash +git add README.md examples/qwp-basic.ts package.json +git commit -m "docs(qwp): document ws:// support and release 4.3.0" +``` + +--- + +## Self-Review + +**1. Spec coverage.** CRC32C (Task 1 — 8.2). Segment format + torn tail (Task 2 — 8.1.5). FSN model, ring, trim, liveness floor (Task 3 — 8.1.1, 8.1.3, 8.1.4). Replay (Task 4 — closes the gap Plan 3 flagged). Boundary records (Task 5 — 8.2). Persisted dictionary, no-dedup recovery (Task 6 — 8.1.6). Delta fallback (Task 7 — 5.2). Slot locks (Task 8 — 8.3). Crash tests (Task 9 — 10 tier 4). Docs and release (Task 10). + +**Knowingly reduced in scope, and why.** Three items are specified but implemented in a simplified form; each is called out so a reviewer sees the gap rather than assuming parity: + +- **Orphan scan and background drainers (8.4)** — Task 8 provides the lock and Task 9 proves a slot survives, but automatic adoption of *another* process's slot is not built. `drain_orphans` defaults to off (9.1), so this matches the default behaviour; enabling it should throw "not implemented" rather than silently doing nothing. +- **Quarantine, rename plus `.failed` sentinel, and the 64-copy cap (8.4)** — not built. A corrupt slot currently fails to open loudly instead of being set aside. +- **`sf_durability=periodic` fsync cadence (8.2)** — the boundary records are crash-safe by construction, but the periodic background barrier is not scheduled; only `memory` durability is wired. + +**2. Placeholder scan.** One deliberate placeholder in Task 9 Step 1's second test, replaced in Step 3 of the same task — flagged inline rather than left silent. + +**3. Type consistency.** `crc32c` (Task 1) is used in Tasks 2, 5, 6. `scanSegment`/`appendFrame` (Task 2) feed `SegmentRing.recovered` (Task 3). `SegmentRing.framesFrom/acknowledge/ackedFsn` (Task 3) are called in Task 4. `readBoundary`/`writeBoundary` (Task 5) and `decodeDictFile` (Task 6) are consumed by the engine in Task 9. `SymbolDict.addRecovered` comes from Plan 2 Task 6 and is relied on in Task 6 here. diff --git a/docs/superpowers/plans/README.md b/docs/superpowers/plans/README.md new file mode 100644 index 0000000..ecaae7f --- /dev/null +++ b/docs/superpowers/plans/README.md @@ -0,0 +1,55 @@ +# QWP implementation plans — read this first + +Four plans implement QWP ingest over `ws://` in this client. They are **strictly +sequential**: each consumes interfaces the previous one produced. + +| Order | Plan | Spec PRs | Deliverable | +|---|---|---|---| +| 1 | `2026-08-07-qwp-plan-1-walking-skeleton.md` | 1–3 | `ws://` ingest works end-to-end, testcontainers-green | +| 2 | `2026-08-07-qwp-plan-2-full-codec.md` | 4–8 | All column types, symbol dictionary, Gorilla, commit frame, cap-splitting | +| 3 | `2026-08-07-qwp-plan-3-errors-and-failover.md` | 9–11 | Response decoding, error policy, poison detector, reconnect, multi-host | +| 4 | `2026-08-07-qwp-plan-4-store-and-forward.md` | 12–16 | Durable send log, replay, crash recovery, release 4.3.0 | + +**Design spec:** `../specs/2026-08-07-qwp-nodejs-client-design.md`. Every plan +cites it by section number; when a plan and the spec disagree, the spec wins and +the plan should be corrected. + +## Things that will bite, in order of likelihood + +These are the traps the spec review surfaced. Each is a case where the *obvious* +implementation is the inverse of the correct one. + +1. **Values are compacted.** A column payload carries only non-null values, not + `rowCount` slots. Getting this wrong yields frames that are self-consistent in + length but wrong in content — the server may accept them and land corrupt + data rather than NACK (spec 6.2.1). +2. **`seq` is not an FSN.** The ACK sequence is connection-scoped and restarts at + 0 on every reconnect. Storing it as an FSN works until the first reconnect, + then trims from near the start of the log (spec 6.6.1). +3. **Gorilla prefixes are bit-reversed.** Packing is LSB-first, so logical `'10'` + is written as `0b01`. Getting it wrong produces plausible-but-wrong + timestamps, not a decode failure (spec 6.3.2). +4. **The notification inbox drops the OLDEST.** Drop-newest is the intuitive + bounded-queue policy and inverts the intent (spec 4.2). +5. **`421` retries forever, `401` never does.** Both are "the server refused the + upgrade"; conflating them either spins on bad credentials or kills a sender + during an ordinary failover (spec 6.5.1). +6. **Decimal scale rescales, it does not reject.** A lock-and-reject port rejects + data Java accepts, order-dependently (spec 6.5.3). +7. **Poison escalation needs strikes AND dwell.** Count alone turns a brief + outage into a producer-fatal terminal (spec 7.4). +8. **A `ws::` sender can silently fall back to ILP v1.** `createBuffer` must + branch on protocol *before* `protocol_version` (spec 3.5). +9. **Size suffixes are 1024-based.** `auto_flush_bytes=64m` read by `parseInt` is + 64 bytes — a flush per row, silently (spec 9.1.1). +10. **Never zero a torn tail during the scan.** It can hold valid-CRC frames that + are the only surviving copy (spec 8.1.5). + +## Scope boundaries worth knowing before you start + +- **No ingest-side compression.** `FLAG_ZSTD` is egress-only; the ingest encoder + never sets it. Do not implement zstd, and the Node version-floor question does + not arise (spec 9.3). +- **The ingest sender is zone-blind.** Endpoint selection ranks by host *state* + only; zone tiers and `target=primary` belong to the query client (spec 1.2). +- **No query client, facade, or UDP sender** in any of these plans (spec 1.1). From 9586a6abfab615da21faf8aa7496e8adcda52f8f Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:01:24 +0100 Subject: [PATCH 036/121] feat(qwp): add unsigned LEB128 varint codec --- src/qwp/protocol/varint.ts | 43 ++++++++++++++++++++++++++++++++++++++ test/qwp/varint.test.ts | 29 +++++++++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 src/qwp/protocol/varint.ts create mode 100644 test/qwp/varint.test.ts diff --git a/src/qwp/protocol/varint.ts b/src/qwp/protocol/varint.ts new file mode 100644 index 0000000..340c0d7 --- /dev/null +++ b/src/qwp/protocol/varint.ts @@ -0,0 +1,43 @@ +import { Buffer } from "node:buffer"; + +/** Unsigned LEB128. 7 data bits per byte; high bit set means another byte follows. */ +export function writeVarint(buf: Buffer, offset: number, value: number): number { + if (value < 0 || !Number.isInteger(value)) { + throw new Error(`varint requires a non-negative integer, got ${value}`); + } + let v = value; + let o = offset; + while (v >= 0x80) { + buf[o++] = (v & 0x7f) | 0x80; + v = Math.floor(v / 128); + } + buf[o++] = v; + return o; +} + +export function varintSize(value: number): number { + let v = value; + let n = 1; + while (v >= 0x80) { + v = Math.floor(v / 128); + n++; + } + return n; +} + +export function readVarint( + buf: Buffer, + offset: number, +): { value: number; offset: number } { + let value = 0; + let shift = 1; + let o = offset; + for (;;) { + if (o >= buf.length) throw new Error("incomplete varint"); + const b = buf[o++]; + value += (b & 0x7f) * shift; + if ((b & 0x80) === 0) break; + shift *= 128; + } + return { value, offset: o }; +} diff --git a/test/qwp/varint.test.ts b/test/qwp/varint.test.ts new file mode 100644 index 0000000..a40afbf --- /dev/null +++ b/test/qwp/varint.test.ts @@ -0,0 +1,29 @@ +import { describe, it, expect } from "vitest"; +import { writeVarint, varintSize, readVarint } from "../../src/qwp/protocol/varint"; + +describe("varint (unsigned LEB128)", () => { + it("encodes single-byte values", () => { + const b = Buffer.alloc(4); + expect(writeVarint(b, 0, 0)).toBe(1); + expect(b[0]).toBe(0x00); + expect(writeVarint(b, 0, 127)).toBe(1); + expect(b[0]).toBe(0x7f); + }); + + it("encodes multi-byte values with the continuation bit", () => { + const b = Buffer.alloc(4); + const end = writeVarint(b, 0, 128); + expect(end).toBe(2); + expect(b[0]).toBe(0x80); + expect(b[1]).toBe(0x01); + }); + + it("round-trips a range of values", () => { + for (const v of [0, 1, 127, 128, 300, 16383, 16384, 1_000_000]) { + const b = Buffer.alloc(10); + const end = writeVarint(b, 0, v); + expect(end).toBe(varintSize(v)); + expect(readVarint(b, 0)).toEqual({ value: v, offset: end }); + } + }); +}); From 7b43966b35a4e2ab6731362527524dbd9f28efaf Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:01:45 +0100 Subject: [PATCH 037/121] feat(qwp): add protocol constants, columnar table buffer, and frame encoder --- src/qwp/protocol/constants.ts | 23 ++++++ src/qwp/protocol/frameEncoder.ts | 134 +++++++++++++++++++++++++++++++ src/qwp/protocol/tableBuffer.ts | 92 +++++++++++++++++++++ test/qwp/constants.test.ts | 19 +++++ test/qwp/frameEncoder.test.ts | 46 +++++++++++ test/qwp/tableBuffer.test.ts | 35 ++++++++ 6 files changed, 349 insertions(+) create mode 100644 src/qwp/protocol/constants.ts create mode 100644 src/qwp/protocol/frameEncoder.ts create mode 100644 src/qwp/protocol/tableBuffer.ts create mode 100644 test/qwp/constants.test.ts create mode 100644 test/qwp/frameEncoder.test.ts create mode 100644 test/qwp/tableBuffer.test.ts diff --git a/src/qwp/protocol/constants.ts b/src/qwp/protocol/constants.ts new file mode 100644 index 0000000..89c5d9d --- /dev/null +++ b/src/qwp/protocol/constants.ts @@ -0,0 +1,23 @@ +import { Buffer } from "node:buffer"; + +/** ASCII "QWP1"; reads as 0x31505751 when interpreted little-endian. */ +export const QWP_MAGIC = Buffer.from("QWP1", "ascii"); +export const QWP_VERSION = 1; +export const HEADER_SIZE = 12; + +export const FLAG_DEFER_COMMIT = 0x01; +export const FLAG_GORILLA = 0x04; +export const FLAG_DELTA_SYMBOL_DICT = 0x08; + +// Column type codes (spec 6.3). Only the four this plan encodes. +export const TYPE_DOUBLE = 0x07; +export const TYPE_SYMBOL = 0x09; +export const TYPE_TIMESTAMP = 0x0a; +export const TYPE_LONG = 0x05; + +// Limits mirrored from the server (spec 6.4). +export const MAX_COLUMNS_PER_TABLE = 2048; +export const MAX_NAME_LENGTH = 127; +export const MAX_ROWS_PER_TABLE = 1_000_000; + +export const WRITE_PATH = "/write/v4"; diff --git a/src/qwp/protocol/frameEncoder.ts b/src/qwp/protocol/frameEncoder.ts new file mode 100644 index 0000000..359af74 --- /dev/null +++ b/src/qwp/protocol/frameEncoder.ts @@ -0,0 +1,134 @@ +import { Buffer } from "node:buffer"; +import { writeVarint, varintSize } from "./varint"; +import { QwpTableBuffer, ColumnBuffer } from "./tableBuffer"; +import { + HEADER_SIZE, + QWP_MAGIC, + QWP_VERSION, + TYPE_DOUBLE, + TYPE_LONG, + TYPE_SYMBOL, + TYPE_TIMESTAMP, +} from "./constants"; + +function utf8Size(s: string): number { + return Buffer.byteLength(s, "utf8"); +} + +/** varint length + utf8 bytes (spec 6.0 "string"). */ +function writeString(buf: Buffer, offset: number, s: string): number { + const n = utf8Size(s); + let o = writeVarint(buf, offset, n); + buf.write(s, o, "utf8"); + return o + n; +} + +function stringSize(s: string): number { + const n = utf8Size(s); + return varintSize(n) + n; +} + +function columnPayloadSize(col: ColumnBuffer, rowCount: number): number { + const nullCount = col.nulls.filter(Boolean).length; + let n = 1; // nullHeader + if (nullCount > 0) n += Math.ceil(rowCount / 8); + const v = col.values.length; + switch (col.type) { + case TYPE_LONG: + case TYPE_DOUBLE: + case TYPE_TIMESTAMP: + return n + v * 8; + case TYPE_SYMBOL: { + // Inline dictionary: varint dictSize, entries, then a varint index per value. + const dict = [...new Set(col.values as string[])]; + n += varintSize(dict.length); + for (const s of dict) n += stringSize(s); + for (const s of col.values as string[]) n += varintSize(dict.indexOf(s)); + return n; + } + default: + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); + } +} + +function writeColumn( + buf: Buffer, + offset: number, + col: ColumnBuffer, + rowCount: number, +): number { + let o = offset; + const nullCount = col.nulls.filter(Boolean).length; + if (nullCount > 0) { + buf[o++] = 1; + const bytes = Math.ceil(rowCount / 8); + buf.fill(0, o, o + bytes); + for (let i = 0; i < rowCount; i++) { + // bit i set means row i is NULL, LSB-first within each byte (spec 6.2.1) + if (col.nulls[i]) buf[o + (i >>> 3)] |= 1 << (i & 7); + } + o += bytes; + } else { + buf[o++] = 0; + } + + switch (col.type) { + case TYPE_LONG: + case TYPE_TIMESTAMP: + for (const v of col.values) { + buf.writeBigInt64LE(BigInt(v as number | bigint), o); + o += 8; + } + return o; + case TYPE_DOUBLE: + for (const v of col.values) { + buf.writeDoubleLE(v as number, o); + o += 8; + } + return o; + case TYPE_SYMBOL: { + const dict = [...new Set(col.values as string[])]; + o = writeVarint(buf, o, dict.length); + for (const s of dict) o = writeString(buf, o, s); + for (const s of col.values as string[]) o = writeVarint(buf, o, dict.indexOf(s)); + return o; + } + default: + throw new Error(`unsupported QWP column type: 0x${col.type.toString(16)}`); + } +} + +function tableSize(t: QwpTableBuffer): number { + let n = stringSize(t.name) + varintSize(t.rowCount) + varintSize(t.columns.length); + for (const c of t.columns) n += stringSize(c.name) + 1; + for (const c of t.columns) n += columnPayloadSize(c, t.rowCount); + return n; +} + +/** Encodes one QWP v1 message. No flags are set in this plan (spec 6.1). */ +export function encodeFrame(tables: QwpTableBuffer[]): Buffer { + const payloadLen = tables.reduce((a, t) => a + tableSize(t), 0); + const buf = Buffer.allocUnsafe(HEADER_SIZE + payloadLen); + + QWP_MAGIC.copy(buf, 0); + buf.writeUInt8(QWP_VERSION, 4); + buf.writeUInt8(0, 5); // flags + buf.writeUInt16LE(tables.length, 6); + buf.writeUInt32LE(payloadLen, 8); + + let o = HEADER_SIZE; + for (const t of tables) { + o = writeString(buf, o, t.name); + o = writeVarint(buf, o, t.rowCount); + o = writeVarint(buf, o, t.columns.length); + for (const c of t.columns) { + o = writeString(buf, o, c.name); + buf.writeUInt8(c.type, o++); + } + for (const c of t.columns) o = writeColumn(buf, o, c, t.rowCount); + } + if (o !== buf.length) { + throw new Error(`frame size mismatch: wrote ${o}, sized ${buf.length}`); + } + return buf; +} diff --git a/src/qwp/protocol/tableBuffer.ts b/src/qwp/protocol/tableBuffer.ts new file mode 100644 index 0000000..37aa6b4 --- /dev/null +++ b/src/qwp/protocol/tableBuffer.ts @@ -0,0 +1,92 @@ +import { Buffer } from "node:buffer"; +import { MAX_COLUMNS_PER_TABLE, MAX_NAME_LENGTH } from "./constants"; + +export interface ColumnBuffer { + name: string; + type: number; + /** Non-null values only — the wire is compacted (spec 6.2.1). */ + values: (number | bigint | string)[]; + /** One entry per row; true means NULL. */ + nulls: boolean[]; + /** Rows accounted for so far, including nulls. */ + size: number; +} + +export class QwpTableBuffer { + readonly name: string; + private readonly cols: ColumnBuffer[] = []; + private readonly byName = new Map(); + private rows = 0; + + constructor(name: string) { + if (!name) throw new Error("table name cannot be empty"); + if (Buffer.byteLength(name, "utf8") > MAX_NAME_LENGTH) { + throw new Error(`table name too long [maxLength=${MAX_NAME_LENGTH}]`); + } + this.name = name; + } + + get rowCount(): number { + return this.rows; + } + + get columns(): ColumnBuffer[] { + return this.cols; + } + + /** Returns null when the column already holds a value for the in-progress row. */ + getOrCreateColumn(name: string, type: number): ColumnBuffer | null { + if (!name) throw new Error("column name cannot be empty"); + const existing = this.byName.get(name); + if (existing) { + if (existing.type !== type) { + throw new Error( + `Column type mismatch for column '${name}': columnType=${existing.type}, sentType=${type}`, + ); + } + // Already has a value for this row -> first value wins, silently. + if (existing.size > this.rows) return null; + existing.nulls.push(false); + existing.size++; + return existing; + } + if (Buffer.byteLength(name, "utf8") > MAX_NAME_LENGTH) { + throw new Error(`column name too long [maxLength=${MAX_NAME_LENGTH}]`); + } + if (this.cols.length >= MAX_COLUMNS_PER_TABLE) { + throw new Error( + `column count exceeds maximum: ${this.cols.length + 1} (max ${MAX_COLUMNS_PER_TABLE})`, + ); + } + // Back-fill this column as null for every row already closed. + const col: ColumnBuffer = { + name, + type, + values: [], + nulls: new Array(this.rows).fill(true), + size: this.rows, + }; + col.nulls.push(false); + col.size++; + this.cols.push(col); + this.byName.set(name, col); + return col; + } + + /** Closes the row, back-filling a null into every column that was not set. */ + nextRow(): void { + this.rows++; + for (const c of this.cols) { + while (c.size < this.rows) { + c.nulls.push(true); + c.size++; + } + } + } + + reset(): void { + this.cols.length = 0; + this.byName.clear(); + this.rows = 0; + } +} diff --git a/test/qwp/constants.test.ts b/test/qwp/constants.test.ts new file mode 100644 index 0000000..c7c2460 --- /dev/null +++ b/test/qwp/constants.test.ts @@ -0,0 +1,19 @@ +import { describe, it, expect } from "vitest"; +import { QWP_MAGIC, HEADER_SIZE, QWP_VERSION, TYPE_LONG, TYPE_SYMBOL } from "../../src/qwp/protocol/constants"; + +describe("QWP constants", () => { + it("magic reads as 0x31505751 little-endian", () => { + expect(QWP_MAGIC.toString("ascii")).toBe("QWP1"); + expect(QWP_MAGIC.readUInt32LE(0)).toBe(0x31505751); + }); + + it("pins header size and version", () => { + expect(HEADER_SIZE).toBe(12); + expect(QWP_VERSION).toBe(1); + }); + + it("pins the type codes this plan uses", () => { + expect(TYPE_LONG).toBe(0x05); + expect(TYPE_SYMBOL).toBe(0x09); + }); +}); diff --git a/test/qwp/frameEncoder.test.ts b/test/qwp/frameEncoder.test.ts new file mode 100644 index 0000000..675f7f5 --- /dev/null +++ b/test/qwp/frameEncoder.test.ts @@ -0,0 +1,46 @@ +import { describe, it, expect } from "vitest"; +import { encodeFrame } from "../../src/qwp/protocol/frameEncoder"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { TYPE_LONG, HEADER_SIZE } from "../../src/qwp/protocol/constants"; + +describe("encodeFrame", () => { + it("writes a valid 12-byte header", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(7); + t.nextRow(); + const f = encodeFrame([t]); + expect(f.subarray(0, 4).toString("ascii")).toBe("QWP1"); + expect(f.readUInt8(4)).toBe(1); // version + expect(f.readUInt8(5)).toBe(0); // flags: none in this plan + expect(f.readUInt16LE(6)).toBe(1); // tableCount + expect(f.readUInt32LE(8)).toBe(f.length - HEADER_SIZE); // payloadLen excludes header + }); + + it("emits nullHeader 0 and compacted values when there are no nulls", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(2); + t.nextRow(); + const f = encodeFrame([t]); + // ...header, table name "t", rowCount 2, colCount 1, schema "a"+type, then column + // nullHeader is the byte immediately after the schema entry. + const idx = f.indexOf(TYPE_LONG, HEADER_SIZE); + expect(f.readUInt8(idx + 1)).toBe(0); // nullHeader = no nulls + expect(f.readBigInt64LE(idx + 2)).toBe(1n); + expect(f.readBigInt64LE(idx + 10)).toBe(2n); + }); + + it("emits nullHeader 1, an LSB-first bitmap, and only non-null values", () => { + const t = new QwpTableBuffer("t"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.nextRow(); // row 1: "a" not set -> null + const f = encodeFrame([t]); + const idx = f.indexOf(TYPE_LONG, HEADER_SIZE); + expect(f.readUInt8(idx + 1)).toBe(1); // bitmap present + expect(f.readUInt8(idx + 2)).toBe(0b00000010); // bit 1 set -> row 1 is NULL + expect(f.readBigInt64LE(idx + 3)).toBe(1n); // only ONE value, not two + expect(f.length).toBe(idx + 3 + 8); + }); +}); diff --git a/test/qwp/tableBuffer.test.ts b/test/qwp/tableBuffer.test.ts new file mode 100644 index 0000000..1b9a789 --- /dev/null +++ b/test/qwp/tableBuffer.test.ts @@ -0,0 +1,35 @@ +import { describe, it, expect } from "vitest"; +import { QwpTableBuffer } from "../../src/qwp/protocol/tableBuffer"; +import { TYPE_LONG, TYPE_DOUBLE } from "../../src/qwp/protocol/constants"; + +describe("QwpTableBuffer", () => { + it("back-fills nulls so all columns stay equal length", () => { + const t = new QwpTableBuffer("trades"); + t.getOrCreateColumn("a", TYPE_LONG)!.values.push(1); + t.nextRow(); + t.getOrCreateColumn("b", TYPE_DOUBLE)!.values.push(2.5); + t.nextRow(); + expect(t.rowCount).toBe(2); + for (const c of t.columns) expect(c.size).toBe(2); + // "a" is null in row 1, "b" is null in row 0 + expect(t.columns.find((c) => c.name === "a")!.nulls).toEqual([false, true]); + expect(t.columns.find((c) => c.name === "b")!.nulls).toEqual([true, false]); + }); + + it("locks a column's type on first sight", () => { + const t = new QwpTableBuffer("x"); + t.getOrCreateColumn("c", TYPE_LONG); + expect(() => t.getOrCreateColumn("c", TYPE_DOUBLE)).toThrow(/type mismatch/i); + }); + + it("ignores a duplicate column within one row (first value wins)", () => { + const t = new QwpTableBuffer("x"); + t.getOrCreateColumn("c", TYPE_LONG)!.values.push(1); + expect(t.getOrCreateColumn("c", TYPE_LONG)).toBeNull(); + }); + + it("rejects an empty column name", () => { + const t = new QwpTableBuffer("x"); + expect(() => t.getOrCreateColumn("", TYPE_LONG)).toThrow(/empty/i); + }); +}); From 1058ebf73e52917dd45e9780ac31ab397e618a0c Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:03:00 +0100 Subject: [PATCH 038/121] feat(qwp): add websocket masking, frame codec, handshake, and transport socket The socket test's emulated server decodes masked client frames (a real server must unmask client frames; the client-side FrameParser correctly rejects them). --- pnpm-workspace.yaml | 6 ++ src/qwp/protocol/frameEncoder.ts | 2 +- src/qwp/ws/frame.ts | 110 +++++++++++++++++++++++++++++ src/qwp/ws/handshake.ts | 98 ++++++++++++++++++++++++++ src/qwp/ws/mask.ts | 13 ++++ src/qwp/ws/socket.ts | 110 +++++++++++++++++++++++++++++ test/qwp/ws.frame.test.ts | 69 +++++++++++++++++++ test/qwp/ws.handshake.test.ts | 75 ++++++++++++++++++++ test/qwp/ws.mask.test.ts | 24 +++++++ test/qwp/ws.socket.test.ts | 114 +++++++++++++++++++++++++++++++ 10 files changed, 620 insertions(+), 1 deletion(-) create mode 100644 pnpm-workspace.yaml create mode 100644 src/qwp/ws/frame.ts create mode 100644 src/qwp/ws/handshake.ts create mode 100644 src/qwp/ws/mask.ts create mode 100644 src/qwp/ws/socket.ts create mode 100644 test/qwp/ws.frame.test.ts create mode 100644 test/qwp/ws.handshake.test.ts create mode 100644 test/qwp/ws.mask.test.ts create mode 100644 test/qwp/ws.socket.test.ts diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml new file mode 100644 index 0000000..8dda36f --- /dev/null +++ b/pnpm-workspace.yaml @@ -0,0 +1,6 @@ +allowBuilds: + '@swc/core': set this to true or false + cpu-features: set this to true or false + esbuild: set this to true or false + protobufjs: set this to true or false + ssh2: set this to true or false diff --git a/src/qwp/protocol/frameEncoder.ts b/src/qwp/protocol/frameEncoder.ts index 359af74..53b6813 100644 --- a/src/qwp/protocol/frameEncoder.ts +++ b/src/qwp/protocol/frameEncoder.ts @@ -18,7 +18,7 @@ function utf8Size(s: string): number { /** varint length + utf8 bytes (spec 6.0 "string"). */ function writeString(buf: Buffer, offset: number, s: string): number { const n = utf8Size(s); - let o = writeVarint(buf, offset, n); + const o = writeVarint(buf, offset, n); buf.write(s, o, "utf8"); return o + n; } diff --git a/src/qwp/ws/frame.ts b/src/qwp/ws/frame.ts new file mode 100644 index 0000000..abb077f --- /dev/null +++ b/src/qwp/ws/frame.ts @@ -0,0 +1,110 @@ +import { Buffer } from "node:buffer"; +import { newMaskKey, applyMask } from "./mask"; + +export const OPCODE = { + CONT: 0x0, + TEXT: 0x1, + BINARY: 0x2, + CLOSE: 0x8, + PING: 0x9, + PONG: 0xa, +} as const; + +const MAX_CONTROL_PAYLOAD = 125; + +/** Client->server frames are always FIN=1 and always masked (spec 3.2.1). */ +export function encodeClientFrame(opcode: number, payload: Buffer): Buffer { + const len = payload.length; + let headerLen = 2; + if (len >= 65536) headerLen += 8; + else if (len >= 126) headerLen += 2; + + const out = Buffer.allocUnsafe(headerLen + 4 + len); + out[0] = 0x80 | opcode; + if (len < 126) { + out[1] = 0x80 | len; + } else if (len < 65536) { + out[1] = 0x80 | 126; + out.writeUInt16BE(len, 2); + } else { + out[1] = 0x80 | 127; + out.writeBigUInt64BE(BigInt(len), 2); + } + const key = newMaskKey(); + key.copy(out, headerLen); + payload.copy(out, headerLen + 4); + applyMask(out.subarray(headerLen + 4), key); + return out; +} + +export class FrameParser { + private buf: Buffer = Buffer.alloc(0); + private fragOpcode = -1; + private frags: Buffer[] = []; + + push(chunk: Buffer): void { + this.buf = this.buf.length === 0 ? chunk : Buffer.concat([this.buf, chunk]); + } + + /** Returns the next complete message, or null when more bytes are needed. */ + next(): { opcode: number; payload: Buffer } | null { + for (;;) { + if (this.buf.length < 2) return null; + const b0 = this.buf[0]; + const b1 = this.buf[1]; + + if ((b0 & 0x70) !== 0) throw new Error("websocket: non-zero RSV bits"); + if ((b1 & 0x80) !== 0) throw new Error("websocket: inbound frame must not be masked"); + + const fin = (b0 & 0x80) !== 0; + const opcode = b0 & 0x0f; + const isControl = (opcode & 0x08) !== 0; + + let len = b1 & 0x7f; + let offset = 2; + if (len === 126) { + if (this.buf.length < 4) return null; + len = this.buf.readUInt16BE(2); + offset = 4; + } else if (len === 127) { + if (this.buf.length < 10) return null; + const big = this.buf.readBigUInt64BE(2); + if (big > BigInt(Number.MAX_SAFE_INTEGER)) throw new Error("websocket: frame too large"); + len = Number(big); + offset = 10; + } + + if (isControl) { + if (len > MAX_CONTROL_PAYLOAD) { + throw new Error(`websocket: control frame payload exceeds ${MAX_CONTROL_PAYLOAD}`); + } + if (!fin) throw new Error("websocket: control frame must not be fragmented"); + } + + if (this.buf.length < offset + len) return null; + const payload = Buffer.from(this.buf.subarray(offset, offset + len)); + this.buf = this.buf.subarray(offset + len); + + // Control frames are never fragmented and interleave freely. + if (isControl) return { opcode, payload }; + + if (opcode === OPCODE.CONT) { + if (this.fragOpcode === -1) throw new Error("websocket: continuation without start"); + this.frags.push(payload); + if (!fin) continue; + const full = Buffer.concat(this.frags); + const op = this.fragOpcode; + this.frags = []; + this.fragOpcode = -1; + return { opcode: op, payload: full }; + } + + if (!fin) { + this.fragOpcode = opcode; + this.frags = [payload]; + continue; + } + return { opcode, payload }; + } + } +} diff --git a/src/qwp/ws/handshake.ts b/src/qwp/ws/handshake.ts new file mode 100644 index 0000000..dc57ea7 --- /dev/null +++ b/src/qwp/ws/handshake.ts @@ -0,0 +1,98 @@ +import { Buffer } from "node:buffer"; +import { createHash, randomBytes } from "node:crypto"; +import { WRITE_PATH } from "../protocol/constants"; + +const WS_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + +export type UpgradeFailureKind = "role-reject" | "auth" | "other"; + +export class QwpUpgradeError extends Error { + readonly status: number; + readonly kind: UpgradeFailureKind; + /** 421 role rejects retry indefinitely; auth failures never do (spec 6.5.1). */ + readonly retriable: boolean; + readonly role?: string; + + constructor(status: number, kind: UpgradeFailureKind, message: string, role?: string) { + super(message); + this.name = "QwpUpgradeError"; + this.status = status; + this.kind = kind; + this.retriable = kind === "role-reject"; + this.role = role; + } +} + +export interface UpgradeResult { + accept: string; + qwpVersion?: number; + maxBatchSize?: number; + role?: string; + /** Bytes already received after the header terminator. */ + leftover: Buffer; +} + +export function computeAccept(key: string): string { + return createHash("sha1").update(key + WS_GUID, "ascii").digest("base64"); +} + +export function buildUpgradeRequest(opts: { + host: string; + port: number; + clientId: string; + authorization?: string; +}): { request: Buffer; key: string } { + const key = randomBytes(16).toString("base64"); + const lines = [ + `GET ${WRITE_PATH} HTTP/1.1`, + `Host: ${opts.host}:${opts.port}`, + "Connection: Upgrade", + "Upgrade: websocket", + "Sec-WebSocket-Version: 13", + `Sec-WebSocket-Key: ${key}`, + "X-QWP-Max-Version: 1", + `X-QWP-Client-Id: ${opts.clientId}`, + ]; + if (opts.authorization) lines.push(`Authorization: ${opts.authorization}`); + return { request: Buffer.from(lines.join("\r\n") + "\r\n\r\n", "ascii"), key }; +} + +export function parseUpgradeResponse(raw: Buffer): UpgradeResult { + const end = raw.indexOf("\r\n\r\n"); + if (end < 0) throw new Error("incomplete HTTP upgrade response"); + const head = raw.subarray(0, end).toString("ascii"); + const leftover = Buffer.from(raw.subarray(end + 4)); + + const [statusLine, ...headerLines] = head.split("\r\n"); + const status = Number.parseInt(statusLine.split(" ")[1], 10); + + const headers = new Map(); + for (const line of headerLines) { + const i = line.indexOf(":"); + if (i > 0) headers.set(line.slice(0, i).trim().toLowerCase(), line.slice(i + 1).trim()); + } + + if (status !== 101) { + const role = headers.get("x-questdb-role"); + if (status === 421 && role) { + throw new QwpUpgradeError(status, "role-reject", `node cannot accept writes [role=${role}]`, role); + } + if (status === 401 || status === 403) { + throw new QwpUpgradeError(status, "auth", `authentication failed [status=${status}]`); + } + throw new QwpUpgradeError(status, "other", `websocket upgrade failed [status=${status}]`); + } + + const accept = headers.get("sec-websocket-accept"); + if (!accept) throw new Error("upgrade response missing Sec-WebSocket-Accept"); + + const version = headers.get("x-qwp-version"); + const cap = headers.get("x-qwp-max-batch-size"); + return { + accept, + qwpVersion: version ? Number.parseInt(version, 10) : undefined, + maxBatchSize: cap ? Number.parseInt(cap, 10) : undefined, + role: headers.get("x-questdb-role"), + leftover, + }; +} diff --git a/src/qwp/ws/mask.ts b/src/qwp/ws/mask.ts new file mode 100644 index 0000000..c6cdd93 --- /dev/null +++ b/src/qwp/ws/mask.ts @@ -0,0 +1,13 @@ +import { Buffer } from "node:buffer"; +import { randomFillSync } from "node:crypto"; + +/** RFC 6455 §10.3 requires a fresh, unpredictable key per frame. */ +export function newMaskKey(): Buffer { + return randomFillSync(Buffer.allocUnsafe(4)); +} + +export function applyMask(payload: Buffer, key: Buffer): void { + for (let i = 0; i < payload.length; i++) { + payload[i] ^= key[i & 3]; + } +} diff --git a/src/qwp/ws/socket.ts b/src/qwp/ws/socket.ts new file mode 100644 index 0000000..bbccaea --- /dev/null +++ b/src/qwp/ws/socket.ts @@ -0,0 +1,110 @@ +import { Buffer } from "node:buffer"; +import { connect as netConnect, Socket } from "node:net"; +import { connect as tlsConnect } from "node:tls"; +import { encodeClientFrame, FrameParser, OPCODE } from "./frame"; +import { buildUpgradeRequest, computeAccept, parseUpgradeResponse } from "./handshake"; + +export interface QwpWebSocketOptions { + host: string; + port: number; + tls: boolean; + clientId: string; + authorization?: string; + rejectUnauthorized?: boolean; + ca?: Buffer | Buffer[]; +} + +export class QwpWebSocket { + private readonly socket: Socket; + private readonly parser = new FrameParser(); + private closed = false; + readonly maxBatchSize?: number; + + private constructor(socket: Socket, maxBatchSize?: number) { + this.socket = socket; + this.maxBatchSize = maxBatchSize; + this.socket.on("data", (chunk: Buffer) => this.onData(chunk)); + } + + static connect(opts: QwpWebSocketOptions): Promise { + return new Promise((resolve, reject) => { + const socket: Socket = opts.tls + ? tlsConnect({ + host: opts.host, + port: opts.port, + rejectUnauthorized: opts.rejectUnauthorized !== false, + ca: opts.ca, + }) + : netConnect({ host: opts.host, port: opts.port }); + + const onError = (e: Error) => reject(e); + socket.once("error", onError); + + socket.once(opts.tls ? "secureConnect" : "connect", () => { + const { request, key } = buildUpgradeRequest(opts); + socket.write(request); + + let acc = Buffer.alloc(0); + const onHeaderData = (chunk: Buffer) => { + acc = Buffer.concat([acc, chunk]); + if (acc.indexOf("\r\n\r\n") < 0) return; + socket.off("data", onHeaderData); + socket.off("error", onError); + try { + const res = parseUpgradeResponse(acc); + if (res.accept !== computeAccept(key)) { + throw new Error("websocket: Sec-WebSocket-Accept mismatch"); + } + const ws = new QwpWebSocket(socket, res.maxBatchSize); + if (res.leftover.length > 0) ws.onData(res.leftover); + resolve(ws); + } catch (e) { + socket.destroy(); + reject(e); + } + }; + socket.on("data", onHeaderData); + }); + }); + } + + private onData(chunk: Buffer): void { + this.parser.push(chunk); + for (let m = this.parser.next(); m; m = this.parser.next()) { + switch (m.opcode) { + case OPCODE.PING: + this.socket.write(encodeClientFrame(OPCODE.PONG, m.payload)); + break; + case OPCODE.CLOSE: + // RFC 6455 §5.5.1: echo the close before tearing down. + if (!this.closed) { + this.closed = true; + this.socket.write(encodeClientFrame(OPCODE.CLOSE, m.payload)); + this.socket.end(); + } + break; + default: + // Response frames are decoded in a later plan (ACK handling). + break; + } + } + } + + /** One write per frame, so a control frame can never interleave mid-frame. */ + sendBinary(payload: Buffer): Promise { + return new Promise((resolve, reject) => { + if (this.closed) return reject(new Error("websocket is closed")); + const frame = encodeClientFrame(OPCODE.BINARY, payload); + this.socket.write(frame, (err) => (err ? reject(err) : resolve())); + }); + } + + close(): Promise { + return new Promise((resolve) => { + if (this.closed) return resolve(); + this.closed = true; + this.socket.write(encodeClientFrame(OPCODE.CLOSE, Buffer.alloc(0))); + this.socket.end(() => resolve()); + }); + } +} diff --git a/test/qwp/ws.frame.test.ts b/test/qwp/ws.frame.test.ts new file mode 100644 index 0000000..5cccd1e --- /dev/null +++ b/test/qwp/ws.frame.test.ts @@ -0,0 +1,69 @@ +import { describe, it, expect } from "vitest"; +import { encodeClientFrame, FrameParser, OPCODE } from "../../src/qwp/ws/frame"; + +/** Server->client frames are never masked (RFC 6455). */ +function serverFrame(opcode: number, payload: Buffer, fin = true): Buffer { + const head: number[] = [(fin ? 0x80 : 0) | opcode]; + if (payload.length < 126) head.push(payload.length); + else if (payload.length < 65536) head.push(126, payload.length >>> 8, payload.length & 0xff); + else throw new Error("test helper: use a small payload"); + return Buffer.concat([Buffer.from(head), payload]); +} + +describe("ws frame codec", () => { + it("encodes a masked client binary frame", () => { + const f = encodeClientFrame(OPCODE.BINARY, Buffer.from([1, 2, 3])); + expect(f[0]).toBe(0x82); // FIN + binary + expect(f[1] & 0x80).toBe(0x80); // mask bit set + expect(f[1] & 0x7f).toBe(3); + expect(f.length).toBe(2 + 4 + 3); + }); + + it("uses the 64-bit length form above 65535", () => { + const f = encodeClientFrame(OPCODE.BINARY, Buffer.alloc(70000)); + expect(f[1] & 0x7f).toBe(127); + expect(Number(f.readBigUInt64BE(2))).toBe(70000); + }); + + it("parses a frame split across chunks", () => { + const whole = serverFrame(OPCODE.BINARY, Buffer.from("abcd")); + const p = new FrameParser(); + p.push(whole.subarray(0, 3)); + expect(p.next()).toBeNull(); + p.push(whole.subarray(3)); + expect(p.next()!.payload.toString()).toBe("abcd"); + }); + + it("defragments continuation frames", () => { + const p = new FrameParser(); + p.push(serverFrame(OPCODE.BINARY, Buffer.from("ab"), false)); + expect(p.next()).toBeNull(); + p.push(serverFrame(OPCODE.CONT, Buffer.from("cd"), true)); + const msg = p.next()!; + expect(msg.opcode).toBe(OPCODE.BINARY); + expect(msg.payload.toString()).toBe("abcd"); + }); + + it("rejects a masked inbound frame", () => { + const f = serverFrame(OPCODE.BINARY, Buffer.from("x")); + f[1] |= 0x80; // claim masked + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/masked/i); + }); + + it("rejects non-zero RSV bits", () => { + const f = serverFrame(OPCODE.BINARY, Buffer.from("x")); + f[0] |= 0x40; + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/rsv/i); + }); + + it("rejects an oversized control frame", () => { + const f = serverFrame(OPCODE.PING, Buffer.alloc(126)); + const p = new FrameParser(); + p.push(f); + expect(() => p.next()).toThrow(/control frame/i); + }); +}); diff --git a/test/qwp/ws.handshake.test.ts b/test/qwp/ws.handshake.test.ts new file mode 100644 index 0000000..4de31bc --- /dev/null +++ b/test/qwp/ws.handshake.test.ts @@ -0,0 +1,75 @@ +import { describe, it, expect } from "vitest"; +import { + buildUpgradeRequest, + computeAccept, + parseUpgradeResponse, + QwpUpgradeError, +} from "../../src/qwp/ws/handshake"; + +describe("qwp handshake", () => { + it("computes Sec-WebSocket-Accept per RFC 6455", () => { + // The canonical example from RFC 6455 §1.3. + expect(computeAccept("dGhlIHNhbXBsZSBub25jZQ==")).toBe("s3pPLMBiTxaQ9kYGzzhZRbK+xOo="); + }); + + it("builds an upgrade request with the QWP headers", () => { + const { request } = buildUpgradeRequest({ host: "h", port: 9000, clientId: "nodejs/1.0.0" }); + const s = request.toString("ascii"); + expect(s).toMatch(/^GET \/write\/v4 HTTP\/1\.1\r\n/); + expect(s).toMatch(/\r\nUpgrade: websocket\r\n/); + expect(s).toMatch(/\r\nSec-WebSocket-Version: 13\r\n/); + expect(s).toMatch(/\r\nX-QWP-Max-Version: 1\r\n/); + expect(s).toMatch(/\r\nX-QWP-Client-Id: nodejs\/1\.0\.0\r\n/); + expect(s.endsWith("\r\n\r\n")).toBe(true); + }); + + it("classifies 421 with a role header as a retriable role reject", () => { + const raw = Buffer.from( + "HTTP/1.1 421 Misdirected Request\r\nX-QuestDB-Role: replica\r\n\r\n", + "ascii", + ); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect(e).toBeInstanceOf(QwpUpgradeError); + expect((e as QwpUpgradeError).kind).toBe("role-reject"); + expect((e as QwpUpgradeError).retriable).toBe(true); + } + }); + + it("classifies 401 as a terminal auth failure", () => { + const raw = Buffer.from("HTTP/1.1 401 Unauthorized\r\n\r\n", "ascii"); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect((e as QwpUpgradeError).kind).toBe("auth"); + expect((e as QwpUpgradeError).retriable).toBe(false); + } + }); + + it("leaves 404 unclassified", () => { + const raw = Buffer.from("HTTP/1.1 404 Not Found\r\n\r\n", "ascii"); + try { + parseUpgradeResponse(raw); + throw new Error("expected throw"); + } catch (e) { + expect((e as QwpUpgradeError).kind).toBe("other"); + } + }); + + it("returns negotiated headers on 101", () => { + const raw = Buffer.from( + "HTTP/1.1 101 Switching Protocols\r\n" + + "Upgrade: websocket\r\nConnection: Upgrade\r\n" + + "Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=\r\n" + + "X-QWP-Version: 1\r\nX-QWP-Max-Batch-Size: 1048576\r\n\r\n", + "ascii", + ); + const r = parseUpgradeResponse(raw); + expect(r.accept).toBe("s3pPLMBiTxaQ9kYGzzhZRbK+xOo="); + expect(r.qwpVersion).toBe(1); + expect(r.maxBatchSize).toBe(1048576); + }); +}); diff --git a/test/qwp/ws.mask.test.ts b/test/qwp/ws.mask.test.ts new file mode 100644 index 0000000..db639b0 --- /dev/null +++ b/test/qwp/ws.mask.test.ts @@ -0,0 +1,24 @@ +import { describe, it, expect } from "vitest"; +import { newMaskKey, applyMask } from "../../src/qwp/ws/mask"; + +describe("ws masking", () => { + it("produces a fresh 4-byte key per call", () => { + const a = newMaskKey(); + const b = newMaskKey(); + expect(a.length).toBe(4); + // Not a strong randomness test; catches a constant/seeded-once key. + const keys = new Set([a.toString("hex"), b.toString("hex")]); + for (let i = 0; i < 20; i++) keys.add(newMaskKey().toString("hex")); + expect(keys.size).toBeGreaterThan(1); + }); + + it("is its own inverse", () => { + const key = Buffer.from([1, 2, 3, 4]); + const original = Buffer.from("hello websocket", "utf8"); + const payload = Buffer.from(original); + applyMask(payload, key); + expect(payload.equals(original)).toBe(false); + applyMask(payload, key); + expect(payload.equals(original)).toBe(true); + }); +}); diff --git a/test/qwp/ws.socket.test.ts b/test/qwp/ws.socket.test.ts new file mode 100644 index 0000000..233ea3c --- /dev/null +++ b/test/qwp/ws.socket.test.ts @@ -0,0 +1,114 @@ +import { describe, it, expect, afterEach } from "vitest"; +import { createServer, Server } from "node:net"; +import { createHash } from "node:crypto"; +import { QwpWebSocket } from "../../src/qwp/ws/socket"; +import { encodeClientFrame, OPCODE } from "../../src/qwp/ws/frame"; + +let server: Server | undefined; +afterEach(() => server?.close()); + +/** + * Decodes client->server frames, which RFC 6455 requires to be MASKED. + * (The client-side FrameParser deliberately rejects masked frames, so it can't + * be reused here — a real server must unmask client frames.) + */ +function maskedFrameDecoder() { + let buf = Buffer.alloc(0); + return function decode(chunk: Buffer): { opcode: number; payload: Buffer }[] { + buf = Buffer.concat([buf, chunk]); + const out: { opcode: number; payload: Buffer }[] = []; + for (;;) { + if (buf.length < 2) break; + const b0 = buf[0]; + const b1 = buf[1]; + const opcode = b0 & 0x0f; + let len = b1 & 0x7f; + let off = 2; + if (len === 126) { + if (buf.length < 4) break; + len = buf.readUInt16BE(2); + off = 4; + } else if (len === 127) { + if (buf.length < 10) break; + len = Number(buf.readBigUInt64BE(2)); + off = 10; + } + const masked = (b1 & 0x80) !== 0; + const keyLen = masked ? 4 : 0; + if (buf.length < off + keyLen + len) break; + const key = buf.subarray(off, off + keyLen); + const payload = Buffer.from(buf.subarray(off + keyLen, off + keyLen + len)); + for (let i = 0; i < payload.length; i++) payload[i] ^= key[i & 3]; + buf = buf.subarray(off + keyLen + len); + out.push({ opcode, payload }); + } + return out; + }; +} + +/** Minimal QWP-ish websocket server: completes the upgrade, echoes nothing. */ +function startServer(onBinary: (b: Buffer) => void): Promise { + return new Promise((resolve) => { + server = createServer((sock) => { + let handshaken = false; + const decode = maskedFrameDecoder(); + sock.on("data", (chunk) => { + if (!handshaken) { + const text = chunk.toString("ascii"); + const key = /Sec-WebSocket-Key: (.+)\r\n/.exec(text)![1]; + const accept = createHash("sha1") + .update(key + "258EAFA5-E914-47DA-95CA-C5AB0DC85B11", "ascii") + .digest("base64"); + sock.write( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n" + + `Connection: Upgrade\r\nSec-WebSocket-Accept: ${accept}\r\n` + + "X-QWP-Version: 1\r\nX-QWP-Max-Batch-Size: 1048576\r\n\r\n", + ); + handshaken = true; + return; + } + for (const m of decode(chunk)) { + if (m.opcode === OPCODE.BINARY) onBinary(m.payload); + if (m.opcode === OPCODE.PING) sock.write(encodeClientFrame(OPCODE.PONG, m.payload)); + } + }); + }); + server.listen(0, "127.0.0.1", () => resolve((server!.address() as any).port)); + }); +} + +describe("QwpWebSocket", () => { + it("connects, negotiates, and sends a binary frame", async () => { + const received: Buffer[] = []; + const port = await startServer((b) => received.push(b)); + const ws = await QwpWebSocket.connect({ + host: "127.0.0.1", + port, + tls: false, + clientId: "nodejs/1.0.0", + }); + expect(ws.maxBatchSize).toBe(1048576); + await ws.sendBinary(Buffer.from("payload")); + await new Promise((r) => setTimeout(r, 50)); + expect(received.length).toBe(1); + expect(received[0].toString()).toBe("payload"); + await ws.close(); + }); + + it("rejects a bad Sec-WebSocket-Accept", async () => { + server = createServer((sock) => { + sock.on("data", () => + sock.write( + "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\n" + + "Connection: Upgrade\r\nSec-WebSocket-Accept: wrong\r\n\r\n", + ), + ); + }); + const port: number = await new Promise((r) => + server!.listen(0, "127.0.0.1", () => r((server!.address() as any).port)), + ); + await expect( + QwpWebSocket.connect({ host: "127.0.0.1", port, tls: false, clientId: "x" }), + ).rejects.toThrow(/accept/i); + }); +}); From 8f0738f0efc40af552f469cdb2115beb482bb2e9 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:03:26 +0100 Subject: [PATCH 039/121] feat(qwp): add QwpBuffer and QwpTransport tls_verify fix vs plan: SenderOptions.tls_verify is parsed to boolean (unsafe_off->false), not a string, so rejectUnauthorized is 'tls_verify !== false'. --- src/qwp/buffer.ts | 127 +++++++++++++++++++++++++++++++++++++ src/qwp/transport.ts | 50 +++++++++++++++ test/qwp/buffer.test.ts | 42 ++++++++++++ test/qwp/transport.test.ts | 15 +++++ 4 files changed, 234 insertions(+) create mode 100644 src/qwp/buffer.ts create mode 100644 src/qwp/transport.ts create mode 100644 test/qwp/buffer.test.ts create mode 100644 test/qwp/transport.test.ts diff --git a/src/qwp/buffer.ts b/src/qwp/buffer.ts new file mode 100644 index 0000000..fb6b35e --- /dev/null +++ b/src/qwp/buffer.ts @@ -0,0 +1,127 @@ +import { Buffer } from "node:buffer"; +import { SenderBuffer } from "../buffer"; +import { TimestampUnit } from "../utils"; +import { QwpTableBuffer } from "./protocol/tableBuffer"; +import { encodeFrame } from "./protocol/frameEncoder"; +import { TYPE_DOUBLE, TYPE_LONG, TYPE_SYMBOL, TYPE_TIMESTAMP } from "./protocol/constants"; + +function toMicros(value: number | bigint, unit: TimestampUnit): bigint { + const v = typeof value === "bigint" ? value : BigInt(Math.trunc(value)); + switch (unit) { + case "ns": + return v / 1000n; + case "ms": + return v * 1000n; + default: + return v; + } +} + +function unsupported(what: string): never { + throw new Error(`${what} is not supported by the QWP buffer yet`); +} + +export class QwpBuffer implements SenderBuffer { + private tables: QwpTableBuffer[] = []; + private byName = new Map(); + private current?: QwpTableBuffer; + private rows = 0; + + reset(): SenderBuffer { + this.tables = []; + this.byName = new Map(); + this.current = undefined; + this.rows = 0; + return this; + } + + table(table: string): SenderBuffer { + let t = this.byName.get(table); + if (!t) { + t = new QwpTableBuffer(table); + this.byName.set(table, t); + this.tables.push(t); + } + this.current = t; + return this; + } + + private require(): QwpTableBuffer { + if (!this.current) throw new Error("table name must be set before adding columns"); + return this.current; + } + + symbol(name: string, value: unknown): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_SYMBOL); + if (col) col.values.push(String(value)); + return this; + } + + intColumn(name: string, value: number): SenderBuffer { + if (!Number.isInteger(value)) throw new Error(`value must be an integer, received ${value}`); + const col = this.require().getOrCreateColumn(name, TYPE_LONG); + if (col) col.values.push(BigInt(value)); + return this; + } + + floatColumn(name: string, value: number): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_DOUBLE); + if (col) col.values.push(value); + return this; + } + + timestampColumn(name: string, value: number | bigint, unit: TimestampUnit = "us"): SenderBuffer { + const col = this.require().getOrCreateColumn(name, TYPE_TIMESTAMP); + if (col) col.values.push(toMicros(value, unit)); + return this; + } + + at(timestamp: number | bigint, unit: TimestampUnit = "us"): void { + const t = this.require(); + const col = t.getOrCreateColumn("timestamp", TYPE_TIMESTAMP); + if (col) col.values.push(toMicros(timestamp, unit)); + t.nextRow(); + this.rows++; + this.current = undefined; + } + + atNow(): void { + const t = this.require(); + t.nextRow(); + this.rows++; + this.current = undefined; + } + + toBufferNew(): Buffer | null { + const dirty = this.tables.filter((t) => t.rowCount > 0); + if (dirty.length === 0) return null; + const frame = encodeFrame(dirty); + this.reset(); + return frame; + } + + toBufferView(): Buffer { + throw new Error("toBufferView is not supported by the QWP buffer"); + } + + currentPosition(): number { + return this.rows; + } + + // Column types arriving in a later plan — fail loudly rather than emit wrong bytes. + stringColumn(): SenderBuffer { + return unsupported("stringColumn"); + } + booleanColumn(): SenderBuffer { + return unsupported("booleanColumn"); + } + arrayColumn(): SenderBuffer { + return unsupported("arrayColumn"); + } + decimalColumnText(): SenderBuffer { + return unsupported("decimalColumnText"); + } + decimalColumn(): SenderBuffer { + return unsupported("decimalColumn"); + } +} diff --git a/src/qwp/transport.ts b/src/qwp/transport.ts new file mode 100644 index 0000000..3dfb74d --- /dev/null +++ b/src/qwp/transport.ts @@ -0,0 +1,50 @@ +import { Buffer } from "node:buffer"; +import { SenderTransport } from "../transport"; +import { SenderOptions } from "../options"; +import { QwpWebSocket } from "./ws/socket"; + +const QWP_DEFAULT_AUTO_FLUSH_ROWS = 1000; // spec 9.1 +const CLIENT_ID = "nodejs/1.0.0"; // protocol client version, not the package version (spec 6.5) + +export class QwpTransport implements SenderTransport { + private readonly options: SenderOptions; + private ws?: QwpWebSocket; + + constructor(options: SenderOptions) { + this.options = options; + } + + async connect(): Promise { + const auth = this.options.username && this.options.password + ? "Basic " + + Buffer.from(`${this.options.username}:${this.options.password}`).toString("base64") + : this.options.token + ? `Bearer ${this.options.token}` + : undefined; + + this.ws = await QwpWebSocket.connect({ + host: this.options.host!, + port: this.options.port!, + tls: this.options.protocol === "wss", + clientId: CLIENT_ID, + authorization: auth, + rejectUnauthorized: this.options.tls_verify !== false, + }); + return true; + } + + async send(data: Buffer): Promise { + if (!this.ws) throw new Error("QWP transport is not connected"); + await this.ws.sendBinary(data); + return true; + } + + async close(): Promise { + await this.ws?.close(); + this.ws = undefined; + } + + getDefaultAutoFlushRows(): number { + return QWP_DEFAULT_AUTO_FLUSH_ROWS; + } +} diff --git a/test/qwp/buffer.test.ts b/test/qwp/buffer.test.ts new file mode 100644 index 0000000..56afa18 --- /dev/null +++ b/test/qwp/buffer.test.ts @@ -0,0 +1,42 @@ +import { describe, it, expect } from "vitest"; +import { QwpBuffer } from "../../src/qwp/buffer"; +import { HEADER_SIZE } from "../../src/qwp/protocol/constants"; + +describe("QwpBuffer", () => { + it("seals a frame containing the buffered rows", () => { + const b = new QwpBuffer(); + b.table("trades").symbol("sym", "ETH").floatColumn("price", 1.5); + b.at(1000n, "us"); + const f = b.toBufferNew()!; + expect(f.subarray(0, 4).toString("ascii")).toBe("QWP1"); + expect(f.readUInt16LE(6)).toBe(1); // one table + expect(f.length).toBeGreaterThan(HEADER_SIZE); + }); + + it("returns null when nothing is buffered", () => { + expect(new QwpBuffer().toBufferNew()).toBeNull(); + }); + + it("accumulates multiple tables into one frame", () => { + const b = new QwpBuffer(); + b.table("a").intColumn("x", 1); + b.at(1n, "us"); + b.table("b").intColumn("y", 2); + b.at(2n, "us"); + expect(b.toBufferNew()!.readUInt16LE(6)).toBe(2); + }); + + it("throws for column types this plan does not encode", () => { + const b = new QwpBuffer(); + b.table("t"); + expect(() => b.booleanColumn("flag", true)).toThrow(/not supported/i); + }); + + it("clears state after sealing", () => { + const b = new QwpBuffer(); + b.table("t").intColumn("x", 1); + b.at(1n, "us"); + b.toBufferNew(); + expect(b.toBufferNew()).toBeNull(); + }); +}); diff --git a/test/qwp/transport.test.ts b/test/qwp/transport.test.ts new file mode 100644 index 0000000..284b06a --- /dev/null +++ b/test/qwp/transport.test.ts @@ -0,0 +1,15 @@ +import { describe, it, expect } from "vitest"; +import { QwpTransport } from "../../src/qwp/transport"; +import { SenderOptions } from "../../src/options"; + +describe("QwpTransport", () => { + it("uses the QWP auto-flush row default, not the ILP one", () => { + const t = new QwpTransport(new SenderOptions("ws::addr=localhost:9000;")); + expect(t.getDefaultAutoFlushRows()).toBe(1000); + }); + + it("refuses to send before connect", async () => { + const t = new QwpTransport(new SenderOptions("ws::addr=localhost:9000;")); + await expect(t.send(Buffer.from([1]))).rejects.toThrow(/not connected/i); + }); +}); From f4f3100f59531b5097b0fff5f46db8f9b927d75e Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:04:55 +0100 Subject: [PATCH 040/121] feat(qwp): route ws:// and wss:// to the QWP buffer and transport --- src/buffer/index.ts | 8 ++++++++ src/index.ts | 2 ++ src/options.ts | 18 ++++++++++++++++-- src/transport/index.ts | 6 +++++- test/qwp/options.test.ts | 38 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 69 insertions(+), 3 deletions(-) create mode 100644 test/qwp/options.test.ts diff --git a/src/buffer/index.ts b/src/buffer/index.ts index 0b61c94..14f6c28 100644 --- a/src/buffer/index.ts +++ b/src/buffer/index.ts @@ -7,11 +7,14 @@ import { PROTOCOL_VERSION_V2, PROTOCOL_VERSION_AUTO, PROTOCOL_VERSION_V3, + WS, + WSS, } from "../options"; import { TimestampUnit } from "../utils"; import { SenderBufferV1 } from "./bufferv1"; import { SenderBufferV2 } from "./bufferv2"; import { SenderBufferV3 } from "./bufferv3"; +import { QwpBuffer } from "../qwp/buffer"; // Default initial buffer size in bytes (64 KB). const DEFAULT_BUFFER_SIZE = 65536; // 64 KB @@ -27,6 +30,11 @@ const DEFAULT_MAX_BUFFER_SIZE = 104857600; // 100 MB * @throws Error if protocol version is not specified or is unsupported */ function createBuffer(options: SenderOptions): SenderBuffer { + // QWP has no protocol_version; this MUST precede the version switch or a + // ws:// sender silently receives SenderBufferV1 and emits ILP text. + if (options.protocol === WS || options.protocol === WSS) { + return new QwpBuffer(); + } switch (options.protocol_version) { case PROTOCOL_VERSION_V3: return new SenderBufferV3(options); diff --git a/src/index.ts b/src/index.ts index bc8e514..3981134 100644 --- a/src/index.ts +++ b/src/index.ts @@ -16,5 +16,7 @@ export { createTransport } from "./transport"; export { TcpTransport } from "./transport/tcp"; export { HttpTransport } from "./transport/http/stdlib"; export { UndiciTransport } from "./transport/http/undici"; +export { QwpBuffer } from "./qwp/buffer"; +export { QwpTransport } from "./qwp/transport"; export type { Logger } from "./logging"; export { bigintToTwosComplementBytes } from "./utils"; diff --git a/src/options.ts b/src/options.ts index f09cc6c..5a94a49 100644 --- a/src/options.ts +++ b/src/options.ts @@ -15,6 +15,8 @@ const HTTP = "http"; const HTTPS = "https"; const TCP = "tcp"; const TCPS = "tcps"; +const WS = "ws"; +const WSS = "wss"; const ON = "on"; const OFF = "off"; @@ -467,16 +469,24 @@ function parseProtocol(options: SenderOptions, configString: string) { case HTTPS: case TCP: case TCPS: + case WS: + case WSS: break; default: throw new Error( - `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps'`, + `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps', 'ws', 'wss'`, ); } return index + 2; } function parseProtocolVersion(options: SenderOptions) { + if (options.protocol === WS || options.protocol === WSS) { + if (options.protocol_version !== undefined && options.protocol_version !== null) { + throw new Error("protocol version is not supported for WebSocket protocol"); + } + return; // stays undefined: createBuffer branches on protocol first + } const protocol_version = options.protocol_version ?? PROTOCOL_VERSION_AUTO; switch (protocol_version) { case PROTOCOL_VERSION_AUTO: @@ -512,6 +522,8 @@ function parseAddress(options: SenderOptions) { switch (options.protocol) { case HTTP: case HTTPS: + case WS: + case WSS: options.port = HTTP_PORT; return; case TCP: @@ -520,7 +532,7 @@ function parseAddress(options: SenderOptions) { return; default: throw new Error( - `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps'`, + `Invalid protocol: '${options.protocol}', accepted protocols: 'http', 'https', 'tcp', 'tcps', 'ws', 'wss'`, ); } } @@ -629,6 +641,8 @@ export { HTTPS, TCP, TCPS, + WS, + WSS, PROTOCOL_VERSION_AUTO, PROTOCOL_VERSION_V1, PROTOCOL_VERSION_V2, diff --git a/src/transport/index.ts b/src/transport/index.ts index 3b0d65d..f370b65 100644 --- a/src/transport/index.ts +++ b/src/transport/index.ts @@ -1,10 +1,11 @@ // @ts-check import { Buffer } from "node:buffer"; -import { SenderOptions, HTTP, HTTPS, TCP, TCPS } from "../options"; +import { SenderOptions, HTTP, HTTPS, TCP, TCPS, WS, WSS } from "../options"; import { UndiciTransport } from "./http/undici"; import { TcpTransport } from "./tcp"; import { HttpTransport } from "./http/stdlib"; +import { QwpTransport } from "../qwp/transport"; /** * Interface for QuestDB transport implementations.
@@ -62,6 +63,9 @@ function createTransport(options: SenderOptions): SenderTransport { case TCP: case TCPS: return new TcpTransport(options); + case WS: + case WSS: + return new QwpTransport(options); default: throw new Error(`Invalid protocol: '${options.protocol}'`); } diff --git a/test/qwp/options.test.ts b/test/qwp/options.test.ts new file mode 100644 index 0000000..45df2b0 --- /dev/null +++ b/test/qwp/options.test.ts @@ -0,0 +1,38 @@ +import { describe, it, expect } from "vitest"; +import { SenderOptions } from "../../src/options"; +import { createBuffer } from "../../src/buffer"; +import { createTransport } from "../../src/transport"; +import { QwpBuffer } from "../../src/qwp/buffer"; +import { QwpTransport } from "../../src/qwp/transport"; + +describe("ws:// wiring", () => { + it("accepts ws:: and defaults the port to 9000", () => { + const o = new SenderOptions("ws::addr=localhost;"); + expect(o.protocol).toBe("ws"); + expect(o.port).toBe(9000); + }); + + it("accepts wss:: ", () => { + expect(new SenderOptions("wss::addr=localhost;").protocol).toBe("wss"); + }); + + it("gives a ws:: sender a QwpBuffer, never an ILP buffer", () => { + const o = new SenderOptions("ws::addr=localhost:9000;"); + expect(createBuffer(o)).toBeInstanceOf(QwpBuffer); + }); + + it("gives a ws:: sender a QwpTransport", () => { + const o = new SenderOptions("ws::addr=localhost:9000;"); + expect(createTransport(o)).toBeInstanceOf(QwpTransport); + }); + + it("rejects protocol_version for ws:: (spec 9.2)", () => { + expect(() => new SenderOptions("ws::addr=localhost:9000;protocol_version=2;")).toThrow( + /not supported for WebSocket/i, + ); + }); + + it("still rejects a genuinely unknown protocol", () => { + expect(() => new SenderOptions("wsx::addr=localhost;")).toThrow(/invalid protocol/i); + }); +}); From 57adecb4fcb9612e191de079a1baf4479123864d Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:05:18 +0100 Subject: [PATCH 041/121] test(qwp): add end-to-end ws:// ingest test against QuestDB Skipped via describe.skipIf when Docker is unavailable so the suite remains portable; runs the real ws:// round-trip against questdb/questdb:nightly when Docker is present. --- test/qwp/integration.test.ts | 71 ++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 test/qwp/integration.test.ts diff --git a/test/qwp/integration.test.ts b/test/qwp/integration.test.ts new file mode 100644 index 0000000..cc3cf44 --- /dev/null +++ b/test/qwp/integration.test.ts @@ -0,0 +1,71 @@ +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { execFileSync } from "node:child_process"; +import { GenericContainer, StartedTestContainer } from "testcontainers"; +import { Sender } from "../../src"; + +function dockerAvailable(): boolean { + try { + execFileSync("docker", ["info"], { stdio: "ignore" }); + return true; + } catch { + return false; + } +} + +let container: StartedTestContainer; +let httpPort: number; + +async function query(sql: string): Promise { + const res = await fetch( + `http://${container.getHost()}:${httpPort}/exec?query=${encodeURIComponent(sql)}`, + ); + return res.json(); +} + +// Skip cleanly when Docker is unavailable so the rest of the suite stays green. +describe.skipIf(!dockerAvailable())("QWP ingest end-to-end", () => { + beforeAll(async () => { + // Matches the existing test/sender.integration.test.ts pattern: no wait + // strategy, readiness is established by the polling loop below. + container = await new GenericContainer("questdb/questdb:nightly") + .withExposedPorts(9000) + .start(); + httpPort = container.getMappedPort(9000); + }, 180_000); + + afterAll(async () => await container?.stop()); + + it("ingests rows over ws:// and they land with correct values", async () => { + // fromConfig is async. Do NOT pass auto_flush=off: spec 9.2 records that + // disabling auto-flush is rejected for WebSocket. The default triggers are + // harmless here because we flush explicitly and then poll. + const sender = await Sender.fromConfig( + `ws::addr=${container.getHost()}:${httpPort};`, + ); + await sender.connect(); + + await sender + .table("qwp_e2e") + .symbol("sym", "ETH-USD") + .floatColumn("price", 2615.54) + .intColumn("qty", 7) + .at(1_700_000_000_000_000n, "us"); + + await sender.flush(); + await sender.close(); + + // WAL apply is asynchronous — poll rather than sleeping a fixed interval. + let rows: any[] = []; + for (let i = 0; i < 60; i++) { + const r = await query("select sym, price, qty from qwp_e2e"); + rows = r.dataset ?? []; + if (rows.length > 0) break; + await new Promise((r) => setTimeout(r, 500)); + } + + expect(rows.length).toBe(1); + expect(rows[0][0]).toBe("ETH-USD"); + expect(rows[0][1]).toBeCloseTo(2615.54, 5); + expect(rows[0][2]).toBe(7); + }, 180_000); +}); From 9d377b042b56d3a44ac0607fc6a7a2ad836b77ca Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:14:09 +0100 Subject: [PATCH 042/121] fix(qwp): encode the designated timestamp as an empty-name column Against a real QuestDB (verified on 10.0.0) the plan's at() emitted a column literally named 'timestamp', which collides with the server's auto-created reserved designated-timestamp column: 'Duplicate column [name=timestamp]' and no rows land. The protocol (source: QwpSchema/QwpTudCache) signals the designated timestamp as a column with an EMPTY schema name (nameLen=0) and TIMESTAMP type; the server names it 'timestamp' and designates it. Only when absent does the server add its own. Verified end-to-end: rows land with sym/price/qty AND our timestamp value. - buffer.at() now uses the empty-name designated timestamp column - QwpTableBuffer allows empty names only for TYPE_TIMESTAMP - regression test pins the empty-name TIMESTAMP schema entry --- src/qwp/buffer.ts | 4 +++- src/qwp/protocol/tableBuffer.ts | 9 +++++++-- test/qwp/buffer.test.ts | 22 +++++++++++++++++++++- test/qwp/integration.test.ts | 4 +++- 4 files changed, 34 insertions(+), 5 deletions(-) diff --git a/src/qwp/buffer.ts b/src/qwp/buffer.ts index fb6b35e..bacb2eb 100644 --- a/src/qwp/buffer.ts +++ b/src/qwp/buffer.ts @@ -78,7 +78,9 @@ export class QwpBuffer implements SenderBuffer { at(timestamp: number | bigint, unit: TimestampUnit = "us"): void { const t = this.require(); - const col = t.getOrCreateColumn("timestamp", TYPE_TIMESTAMP); + // Designated timestamp column: an empty schema name signals the designated + // timestamp (QwpSchema nameLen=0); the server names it "timestamp". + const col = t.getOrCreateColumn("", TYPE_TIMESTAMP); if (col) col.values.push(toMicros(timestamp, unit)); t.nextRow(); this.rows++; diff --git a/src/qwp/protocol/tableBuffer.ts b/src/qwp/protocol/tableBuffer.ts index 37aa6b4..68ad979 100644 --- a/src/qwp/protocol/tableBuffer.ts +++ b/src/qwp/protocol/tableBuffer.ts @@ -1,5 +1,5 @@ import { Buffer } from "node:buffer"; -import { MAX_COLUMNS_PER_TABLE, MAX_NAME_LENGTH } from "./constants"; +import { MAX_COLUMNS_PER_TABLE, MAX_NAME_LENGTH, TYPE_TIMESTAMP } from "./constants"; export interface ColumnBuffer { name: string; @@ -36,7 +36,12 @@ export class QwpTableBuffer { /** Returns null when the column already holds a value for the in-progress row. */ getOrCreateColumn(name: string, type: number): ColumnBuffer | null { - if (!name) throw new Error("column name cannot be empty"); + // An empty name is reserved for the designated timestamp column (TYPE_TIMESTAMP): + // QwpSchema allows nameLen=0 to signal the designated timestamp, and the server + // names it "timestamp". A non-empty name of any other type must not be empty. + if (!name && type !== TYPE_TIMESTAMP) { + throw new Error("column name cannot be empty"); + } const existing = this.byName.get(name); if (existing) { if (existing.type !== type) { diff --git a/test/qwp/buffer.test.ts b/test/qwp/buffer.test.ts index 56afa18..b73148c 100644 --- a/test/qwp/buffer.test.ts +++ b/test/qwp/buffer.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect } from "vitest"; import { QwpBuffer } from "../../src/qwp/buffer"; -import { HEADER_SIZE } from "../../src/qwp/protocol/constants"; +import { HEADER_SIZE, TYPE_LONG, TYPE_TIMESTAMP } from "../../src/qwp/protocol/constants"; describe("QwpBuffer", () => { it("seals a frame containing the buffered rows", () => { @@ -39,4 +39,24 @@ describe("QwpBuffer", () => { b.toBufferNew(); expect(b.toBufferNew()).toBeNull(); }); + + it("writes the designated timestamp as an empty-name TIMESTAMP column", () => { + // QwpSchema allows nameLen=0 for the designated timestamp; the server names + // it "timestamp". A column literally named "timestamp" collides with the + // server's reserved designated column, so it must NOT be emitted. + const b = new QwpBuffer(); + b.table("t").intColumn("x", 1); + b.at(1_700_000_000_000_000n, "us"); + const f = b.toBufferNew()!; + let o = HEADER_SIZE; + o += f[o] + 1; // table name "t" (varint len + bytes) + o += 1; // rowCount + const cc = f[o++]; + expect(cc).toBe(2); + const nx = f[o++]; + o += nx; + expect(f[o++]).toBe(TYPE_LONG); // "x" + expect(f[o++]).toBe(0); // empty column name length -> designated timestamp + expect(f[o++]).toBe(TYPE_TIMESTAMP); + }); }); diff --git a/test/qwp/integration.test.ts b/test/qwp/integration.test.ts index cc3cf44..c3c6536 100644 --- a/test/qwp/integration.test.ts +++ b/test/qwp/integration.test.ts @@ -57,7 +57,7 @@ describe.skipIf(!dockerAvailable())("QWP ingest end-to-end", () => { // WAL apply is asynchronous — poll rather than sleeping a fixed interval. let rows: any[] = []; for (let i = 0; i < 60; i++) { - const r = await query("select sym, price, qty from qwp_e2e"); + const r = await query("select sym, price, qty, timestamp from qwp_e2e"); rows = r.dataset ?? []; if (rows.length > 0) break; await new Promise((r) => setTimeout(r, 500)); @@ -67,5 +67,7 @@ describe.skipIf(!dockerAvailable())("QWP ingest end-to-end", () => { expect(rows[0][0]).toBe("ETH-USD"); expect(rows[0][1]).toBeCloseTo(2615.54, 5); expect(rows[0][2]).toBe(7); + // The designated timestamp lands with OUR value, not receive time. + expect(rows[0][3]).toBe("2023-11-14T22:13:20.000000Z"); }, 180_000); }); From f6582a4e27d9789964e469bf78614f2e8e6c8284 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:16:29 +0100 Subject: [PATCH 043/121] test(qwp): allow the e2e test to target an already-running QuestDB QWP_TEST_ADDR=host:port connects to an existing server (e.g. QWP_TEST_ADDR= localhost:9000) instead of starting a testcontainer; falling back to questdb/questdb:nightly when unset, and skipping when neither Docker nor an external address is available. Uses a per-run table name so re-verifying against a persistent server is idempotent. --- test/qwp/integration.test.ts | 57 +++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 11 deletions(-) diff --git a/test/qwp/integration.test.ts b/test/qwp/integration.test.ts index c3c6536..3197a31 100644 --- a/test/qwp/integration.test.ts +++ b/test/qwp/integration.test.ts @@ -12,40 +12,75 @@ function dockerAvailable(): boolean { } } -let container: StartedTestContainer; -let httpPort: number; +/** + * Resolve the ingest endpoint. Three modes: + * + * 1. `QWP_TEST_ADDR=host:port` (e.g. "localhost:9000") — connect to an + * ALREADY-RUNNING QuestDB instead of starting one. Use this to verify against + * a container you manage yourself, or any external server. + * 2. Otherwise start `questdb/questdb:nightly` via testcontainers. + * 3. If neither is possible (no `QWP_TEST_ADDR` and Docker unavailable), skip. + */ +function parseAddr( + addr: string | undefined, +): { host: string; port: number } | null { + if (!addr) return null; + const idx = addr.lastIndexOf(":"); + if (idx < 0) return { host: addr, port: 9000 }; + return { host: addr.slice(0, idx), port: Number(addr.slice(idx + 1)) }; +} + +const external = parseAddr(process.env.QWP_TEST_ADDR); +const useExternal = external !== null; +const canRun = useExternal || dockerAvailable(); + +let container: StartedTestContainer | undefined; +let host = external?.host ?? "localhost"; +let httpPort = external?.port ?? 0; async function query(sql: string): Promise { const res = await fetch( - `http://${container.getHost()}:${httpPort}/exec?query=${encodeURIComponent(sql)}`, + `http://${host}:${httpPort}/exec?query=${encodeURIComponent(sql)}`, ); return res.json(); } -// Skip cleanly when Docker is unavailable so the rest of the suite stays green. -describe.skipIf(!dockerAvailable())("QWP ingest end-to-end", () => { +describe.skipIf(!canRun)("QWP ingest end-to-end", () => { beforeAll(async () => { + if (useExternal) { + // Confirm the target is actually a reachable QuestDB before proceeding. + const probe = await query("SELECT 1").catch(() => null); + if (!probe || probe.error) { + throw new Error( + `QWP_TEST_ADDR=${process.env.QWP_TEST_ADDR} is not a reachable QuestDB HTTP endpoint`, + ); + } + return; + } // Matches the existing test/sender.integration.test.ts pattern: no wait // strategy, readiness is established by the polling loop below. container = await new GenericContainer("questdb/questdb:nightly") .withExposedPorts(9000) .start(); + host = container.getHost(); httpPort = container.getMappedPort(9000); }, 180_000); - afterAll(async () => await container?.stop()); + afterAll(async () => { + await container?.stop(); + }); it("ingests rows over ws:// and they land with correct values", async () => { + // Unique table per run, so re-verifying against a persistent server is idempotent. + const table = `qwp_e2e_${Date.now()}`; // fromConfig is async. Do NOT pass auto_flush=off: spec 9.2 records that // disabling auto-flush is rejected for WebSocket. The default triggers are // harmless here because we flush explicitly and then poll. - const sender = await Sender.fromConfig( - `ws::addr=${container.getHost()}:${httpPort};`, - ); + const sender = await Sender.fromConfig(`ws::addr=${host}:${httpPort};`); await sender.connect(); await sender - .table("qwp_e2e") + .table(table) .symbol("sym", "ETH-USD") .floatColumn("price", 2615.54) .intColumn("qty", 7) @@ -57,7 +92,7 @@ describe.skipIf(!dockerAvailable())("QWP ingest end-to-end", () => { // WAL apply is asynchronous — poll rather than sleeping a fixed interval. let rows: any[] = []; for (let i = 0; i < 60; i++) { - const r = await query("select sym, price, qty, timestamp from qwp_e2e"); + const r = await query(`select sym, price, qty, timestamp from ${table}`); rows = r.dataset ?? []; if (rows.length > 0) break; await new Promise((r) => setTimeout(r, 500)); From 39e6357fdba5f1608700897b2c7d75c411ef9090 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Fri, 7 Aug 2026 23:18:29 +0100 Subject: [PATCH 044/121] docs: add plan 1 -> plan 2 handoff for the next agent --- .../plans/HANDOFF-plan1-to-plan2.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 docs/superpowers/plans/HANDOFF-plan1-to-plan2.md diff --git a/docs/superpowers/plans/HANDOFF-plan1-to-plan2.md b/docs/superpowers/plans/HANDOFF-plan1-to-plan2.md new file mode 100644 index 0000000..3799bcc --- /dev/null +++ b/docs/superpowers/plans/HANDOFF-plan1-to-plan2.md @@ -0,0 +1,33 @@ +# Handoff — Plan 1 (walking skeleton) done, for the agent executing Plan 2 + +**Read first:** `docs/superpowers/plans/README.md`, then `2026-08-07-qwp-plan-2-full-codec.md`. +The design spec wins over any plan text: `docs/superpowers/specs/2026-08-07-qwp-nodejs-client-design.md`. + +## State + +- Branch `feat/qwp-design` at `f6582a4`. Plan 1 (PRs 1–3) fully implemented and committed (8 task commits). Full suite green: 206 passed; the Docker e2e was verified against a live QuestDB 10.0.0 and passes with `QWP_TEST_ADDR=localhost:9000`. +- New code under `src/qwp/`: `protocol/{varint,constants,tableBuffer,frameEncoder}.ts`, `ws/{mask,frame,handshake,socket}.ts`, `buffer.ts`, `transport.ts`. Wired into `src/{options,buffer/index,transport/index,index}.ts` (`WS`/`WSS`, protocol-first `createBuffer` branch, `QwpBuffer`/`QwpTransport` exports). Tests in `test/qwp/` (incl. a testcontainers e2e). + +## Three corrections to the plan (verified against the real server / Java source) — carry forward + +1. **Designated timestamp = EMPTY-name column (critical).** `at()` emits a column with **`nameLen=0`** and `TYPE_TIMESTAMP`; the server names it `timestamp` and designates it, and only auto-adds its own when none is present. Sending a column literally named `timestamp` collides → `Duplicate column [name=timestamp]`, no rows land. `QwpTableBuffer.getOrCreateColumn` allows an empty name only for `TYPE_TIMESTAMP`. The compiled plan for Plan 2's codec and any golden vectors must treat `nameLen=0` as the designated timestamp. Ground truth: `QwpSchema` / `QwpTudCache` in the local Java clone. +2. **Clients mask, servers unmask.** Client→server WS frames are masked; the client-side `FrameParser` correctly rejects masked frames. The e2e emulated server in `test/qwp/ws.socket.test.ts` was patched to decode masked client frames (a real server must unmask). +3. **`tls_verify` is a `boolean`** in `SenderOptions` (parsed `unsafe_off`→false), not a string — guard TLS as `!== false`. + +## Environment notes + +- **`pnpm