feat(qwp): QWP ingest over WebSocket with store-and-forward durability + benchmarks - #61

Draft

nwoolmer wants to merge 121 commits into

mainfrom

feat/qwp-design

Commits on Aug 7, 2026

docs: design for QWP ingest over ws:// in the Node.js client

nwoolmer
and
claude
committed
dd480f0
docs: correct QWP wire format after line-by-line Java re-review

nwoolmer
and
claude
committed
4e5536c
docs: fix config-key ownership and poison escalation in QWP spec

nwoolmer
and
claude
committed
d46cb6f
docs: fill in QWP store-and-forward internals after fourth review pass

nwoolmer
and
claude
committed
96ab58f
docs: add reconnect dict catch-up and recovery no-dedup rule to QWP spec

nwoolmer
and
claude
committed
6b092f0
docs: add hot-spare provisioning and close() ordering to QWP spec

nwoolmer
and
claude
committed
1c48919
docs: add cap-split flush, dual dict modes and WS control frames to QWP spec

nwoolmer
and
claude
committed
a47fbd6
docs: add reset() watermark rollback and the three callback surfaces

nwoolmer
and
claude
committed
b8a0e40
docs: specify the Gorilla bitstream, including the bit-reversed prefixes

nwoolmer
and
claude
committed
4246912
docs: correct the Node-side integration scope in the QWP spec

nwoolmer
and
claude
committed
56e4dbf
docs: resolve single-endpoint scope contradiction and stale testing claims

nwoolmer
and
claude
committed
bf84b57
docs: specify upgrade-failure classification and TLS trust-store mapping

nwoolmer
and
claude
committed
a2e69e6
docs: add the commit frame, row rollback, and missing wire limits

nwoolmer
and
claude
committed
9ad6d69
docs: cover PR 4/5 column rules and the remaining ingest classes

nwoolmer
and
claude
committed
9cba6dc
docs: describe the staging buffer swap and close the class coverage matrix

nwoolmer
and
claude
committed
b04a237
docs: correct notification inbox policy and the close() suppression test

nwoolmer
and
claude
committed
94193ad
docs: correct per-dispatcher capacities and complete the event kind list

nwoolmer
and
claude
committed
52f643e
docs: add config value grammars and correct the auto_flush_bytes default

nwoolmer
and
claude
committed
63bf938
docs: add cross-key validation and correct the sf_durability mode count

nwoolmer
and
claude
committed
ead61b2
docs: add connect timing and precise sf_durability semantics

nwoolmer
and
claude
committed
26920e5
docs: add quarantine mechanics, the unset sentinel rule, and missing defaults

nwoolmer
and
claude
committed
267aec8
docs: add auth precedence rules and the wss-only TLS constraint

nwoolmer
and
claude
committed
07b303b
docs: bring multi-host failover into scope and add the addr grammar

nwoolmer
and
claude
committed
7c11e57
docs: point-by-point consistency fixes after the failover scope change

nwoolmer
and
claude
committed
536eb65
docs: resolve the staging-buffer decision and renumber the PR stack

nwoolmer
and
claude
committed
5d88829
docs: remove ingest-side zstd, which is an egress-only feature

nwoolmer
and
claude
committed
65b78b0
docs: correct what actually drives endpoint rotation

nwoolmer
and
claude
committed
a14d239
docs: complete the configuration defaults and note durability without recovery

nwoolmer
and
claude
committed
8168a52
docs: verify the golden-vector harness is buildable and record its template

nwoolmer
and
claude
committed
a064c07
docs: specify frame parsing, including inbound defragmentation

nwoolmer
and
claude
committed
6c12996
docs: add ACK-to-FSN correlation and the delta-dict runtime fallback

nwoolmer
and
claude
committed
098cc9c
docs: define the FSN model, which eight other sections depended on

nwoolmer
and
claude
committed
6b6f95a
docs: specify torn-tail segment recovery

nwoolmer
and
claude
committed
8c815ba
docs: add implementation plan 1, the ws:// walking skeleton

nwoolmer
and
claude
committed
eba0ec7
docs: add implementation plans 2 to 4 and a plan index

nwoolmer
and
claude
committed
34b40d1
feat(qwp): add unsigned LEB128 varint codec
nwoolmer
committed
9586a6a
feat(qwp): add protocol constants, columnar table buffer, and frame encoder
nwoolmer
committed
7b43966
feat(qwp): add websocket masking, frame codec, handshake, and transport socket
nwoolmer
committed
1058ebf
feat(qwp): add QwpBuffer and QwpTransport
nwoolmer
committed
8f0738f
feat(qwp): route ws:// and wss:// to the QWP buffer and transport
nwoolmer
committed
f4f3100
test(qwp): add end-to-end ws:// ingest test against QuestDB
nwoolmer
committed
57adecb
fix(qwp): encode the designated timestamp as an empty-name column
nwoolmer
committed
9d377b0
test(qwp): allow the e2e test to target an already-running QuestDB
nwoolmer
committed
f6582a4
docs: add plan 1 -> plan 2 handoff for the next agent
nwoolmer
committed
39e6357
refactor(qwp): allow a flush to produce multiple frames
nwoolmer
committed
c573fb2
feat(qwp): add remaining fixed-width column types
nwoolmer
committed
93cba9f
feat(qwp): add VARCHAR and BINARY columns
nwoolmer
committed
dffcd95
feat(qwp): add array, decimal and geohash columns with their locks
nwoolmer
committed
c2d5a94
feat(qwp): roll back the in-progress row when a setter throws
nwoolmer
committed
c917b42
feat(qwp): add connection-scoped symbol dictionary
nwoolmer
committed
0d83058
feat(qwp): add delta symbol dictionary encoding
nwoolmer
committed
9e8df43
feat(qwp): add Gorilla timestamp encoding
nwoolmer
committed
ac48760
feat(qwp): add cap-splitting and the commit frame
nwoolmer
committed
4f02488
test(qwp): end-to-end coverage for all supported column types
nwoolmer
committed
666fa74
test(qwp): use a still-unsupported setter for the not-supported guard
nwoolmer
committed
c0449d0
docs: add plan 2 -> plan 3 handoff for the next agent
nwoolmer
committed
e336848
feat(qwp): decode server response frames
nwoolmer
committed
90eb9bc
feat(qwp): add error categories and default policy mapping
nwoolmer
committed
8289e8f
feat(qwp): correlate connection-scoped ACK sequences to FSNs
nwoolmer
committed
4378a9d
feat(qwp): handle ACK and NACK responses, surface in-flight loss
nwoolmer
committed
abe782d
feat(qwp): add poison-frame detector with strike and dwell conditions
nwoolmer
committed
8a62b98
feat(qwp): parse multi-host addr lists with IPv6 support
nwoolmer
committed
fcf7b37
feat(qwp): add state-ranked host tracker with private cursors
nwoolmer
committed
72281a2
feat(qwp): add reconnect, endpoint rotation and dictionary catch-up
nwoolmer
committed
9dfee9b
feat(qwp): add drop-oldest dispatchers and derived connect mode
nwoolmer
committed
0db1b2c
fix(qwp): enforce spec config minimum of 16 for notification inbox capacities
nwoolmer
committed
8b03974
docs: add plan 3 -> plan 4 handoff for the next agent
nwoolmer
committed
b5399b2
feat(qwp): add CRC32C (Castagnoli) implementation
nwoolmer
committed
5995104
feat(qwp): add SF01 segment format with torn-tail recovery
nwoolmer
committed
b5f2b6c
feat(qwp): add FSN-keyed segment ring with ACK-driven trim
nwoolmer
committed
e6b271e
feat(qwp): replay unacked frames from the ring after reconnect
nwoolmer
committed
9b9c59e
feat(qwp): add alternating-generation crash-safe boundary record
nwoolmer
committed
4ff4a2f
feat(qwp): add SYD1 persisted symbol dictionary
nwoolmer
committed
68cb938
feat(qwp): degrade to full-dict mode when the symbol side file fails
nwoolmer
committed
0250872
feat(qwp): add slot locking and the store-and-forward engine
nwoolmer
committed
36daae2
test(qwp): add crash-recovery tests for store-and-forward
nwoolmer
committed
d183eb3
feat(qwp): wire the store-and-forward engine into the transport
nwoolmer
committed
f878edd
docs(qwp): document ws:// support and release 4.3.0
nwoolmer
committed
34a9ed6
fix(qwp): review fixes for segment clobbering, delta sizing, boundary magic, durability docs
nwoolmer
committed
e5f6876
feat(qwp): periodic durability, quarantine, durable acks, and connect/ack fixes
nwoolmer
committed
b378dbc
feat(qwp): wire the symbol dictionary into the buffer for delta mode (B1)
nwoolmer
committed
f5fa8d2
feat(qwp): orphan scan + background drainers to replay crashed slots (A1)
nwoolmer
committed
b98cbea
docs(qwp): handoff plan7 — A1 orphan drainers complete
nwoolmer
committed
ddedf4b
feat(qwp): sf-manifest.bin chain-head manifest for recovery cross-check (C2)
nwoolmer
committed
6b46c43
docs(qwp): handoff plan8 — C2 manifest complete
nwoolmer
committed
2d3ddb7

Commits on Aug 8, 2026

docs: benchmark suite design for the QWP Node client

nwoolmer
and
claude
committed
f9e2fbb
docs: add Plan B, the QWP benchmark suite

nwoolmer
and
claude
committed
c8c367e
docs: fix four defects in Plan B found by checking it against the code

nwoolmer
and
claude
committed
4c34fce
docs: second review pass on Plan B — three more defects

nwoolmer
and
claude
committed
cf18eee
docs: third review pass on Plan B — verified the harness by running it

nwoolmer
and
claude
committed
f9461ec
docs: fourth review pass on Plan B — reorder validation before measurement

nwoolmer
and
claude
committed
a0f6400
chore: release 5.0.0 instead of 4.3.0, with the rationale recorded

nwoolmer
and
claude
committed
622cdad
docs: fifth review pass on Plan B — priming the dict does not enable delta

nwoolmer
and
claude
committed
d72ccbc
docs: sixth review pass on Plan B — Tasks 7 to 9 line by line

nwoolmer
and
claude
committed
7540628
docs: seventh review pass — Task 9 README and a spec-to-plan sweep

nwoolmer
and
claude
committed
84fdd3b
docs: eighth review pass — sinks, and a Self-Review that can catch gaps

nwoolmer
and
claude
committed
ef9b4f2
docs: review passes nine to thirteen — ten defects, then a clean pass

nwoolmer
and
claude
committed
118b45d
docs: review passes fourteen to eighteen — six defects

nwoolmer
and
claude
committed
2c95693
docs: review passes nineteen to twenty-three — seven defects

nwoolmer
and
claude
committed
aa26ba1
docs: review passes twenty-four to twenty-eight — the benchmark tree is unchecked

nwoolmer
and
claude
committed
0e98f46
docs: review passes twenty-nine to thirty-three — one defect, four clean

nwoolmer
and
claude
committed
68c8237
docs: review passes thirty-four to thirty-eight — widened beyond Plan B

nwoolmer
and
claude
committed
efb78d0
docs: review passes thirty-nine to forty-three — the handoff chain

nwoolmer
and
claude
committed
ef61af3
docs: review passes forty-four to forty-eight — plan vs current source

nwoolmer
and
claude
committed
d652cf4
docs: review passes forty-nine to fifty-three — auditing "nothing is deferred"

nwoolmer
and
claude
committed
21a75ff
docs: review passes fifty-four to fifty-eight — spec 9.1 defaults vs shipped

nwoolmer
and
claude
committed
7384910
docs: review passes fifty-nine to sixty-three — four spec-9.1 gaps in shipped code

nwoolmer
and
claude
committed
ea28b7a
docs: review passes sixty-four to sixty-eight — wire format holds, one unenforced limit

nwoolmer
and
claude
committed
50afb9f
bench: add seeded workload generators
nwoolmer
committed
2a12dc6
bench: add hand-written floor baselines
nwoolmer
committed
f04cd0d
bench: add bench scripts and split bench files from tests
nwoolmer
committed
48bd28b
test: add wire-format invariants over the benchmark workloads
nwoolmer
committed
712a0e1
bench: add encoder throughput benchmarks with floor comparison
nwoolmer
committed
2ad0e88
bench: add row-building benchmarks for QwpBuffer
nwoolmer
committed
519fd79
bench: add store-and-forward benchmarks with a disk baseline
nwoolmer
committed
b55ab21
bench: add end-to-end latency script for both flush contracts
nwoolmer
committed
da35467
docs: how to run the QWP benchmarks and how to read them
nwoolmer
committed
171a18e
chore: remove internal QWP planning docs (plans and specs) from the public branch
nwoolmer
committed
131d46b
fix(qwp): close five spec 9.1/6.4/7.4 gaps in the shipped client
nwoolmer
committed
215b44f
fix(qwp): address review data safety findings
nwoolmer
committed
f41da02
fix(qwp): harden rereview failure paths
nwoolmer
committed
9ffdc6c