Add Durable Objects realtime control plane - #92
Draft
XeonFX wants to merge 107 commits into
Draft
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
peerly | e49cc4e | Commit Preview URL Branch Preview URL |
Aug 03 2026, 05:58 AM |
added 4 commits
July 23, 2026 10:11
Adds the companion implementation guide with exact file layout, wire protocol, SQL schemas, crypto recipes, and PR sequence, and fixes the shard-fan-out cost bug in the stats/directory design.
Implements the shared v1 wire protocol, HMAC capability/cookie auth, UserGatewayDO (hibernatable control socket, seek reservation, event resume), SignalScopeDO (opaque WebRTC signaling relay), and WorkspaceDO (membership/presence coordination) from docs/DURABLE_OBJECTS_ARCHITECTURE.md and docs/DURABLE_OBJECTS_IMPLEMENTATION.md. Also adds InterestQueueDO, PresenceStatsShardDO, and RoomDirectoryShardDO to @peerly/core/worker/realtime — HeyHubs-only classes that live here so both apps share one implementation and one set of tests. Production stays on COORDINATION_BACKEND=legacy-relay; only the peerly-preview environment flips to durable-objects for pre-cutover testing. No existing signaling/workspace path is touched yet — this is additive infrastructure per the Phase 1 exit criteria.
Both were written earlier in this branch's history but never staged — closing that gap before rebasing onto main.
XeonFX
force-pushed
the
durable-objects-control-plane
branch
from
July 23, 2026 08:29
9f02a79 to
05969a5
Compare
added 2 commits
July 23, 2026 10:43
Workers Builds deploys non-production branches with `wrangler versions upload`, and Cloudflare rejects any version carrying a Durable Object migration (API error 10211) — so the migrations block in wrangler.jsonc turned every branch build red and broke branch preview aliases. Moves all DO configuration (bindings, migrations, realtime secrets) into a standalone wrangler.preview.jsonc that deploys the peerly-preview staging worker and backs vitest-pool-workers. The default config returns to production parity plus the inert COORDINATION_BACKEND=legacy-relay var; migrations enter wrangler.jsonc only in the Phase 5 cutover PR, where merge-to-main applies them via a full deploy. Also: test:workers now creates dist/ first (vitest-pool-workers refuses to start when the assets directory is missing on a fresh CI checkout), and authorize/publish/join no longer overwrite an earlier pending alarm with a later expiry, which could delay pruning.
Owner
Author
|
Stable Durable Objects preview: https://preview.peerly.cc Deployed from |
Owner
Author
|
Preview auth build fixed and redeployed: https://preview.peerly.cc The preview Wrangler config now injects the public Google client ID during the Vite build, preventing future deploys from compiling out the identity provider. Live UI verified at |
added 19 commits
July 23, 2026 12:19
- Implement InterestQueueDO/PresenceStatsShardDO/RoomDirectoryShardDO in the HeyHubs repo instead of packages/core: each has exactly one consumer (HeyHubs never binds them), so keeping them in core was Peerly source carrying HeyHubs product code with no shared-code benefit. - Close version mismatches with 4002 instead of leaving the envelope-level case mapped to 4003 and the hello-negotiation case not closing at all. - Enforce the Origin allowlist and a request-size cap on /api/network/enroll and /api/network/session, matching the control/signal upgrade paths. - Give room-directory entries their own TTL instead of borrowing the matchmaking seek lease. - Re-check the interest-length bound after NFKC normalization, which can expand a string past the protocol-layer pre-normalization check. - Time out a RealtimeClient command that never gets an ack/error, and fail in-flight commands immediately on an explicit close() instead of letting them expire on their own. - Add a cross-file consistency test so CLIENT_LIMITS can't silently drift from the server LIMITS it mirrors. - Regenerate a second types file (worker-configuration.preview.d.ts) for the DO-bound preview config and wire it into worker:check/CI. See docs/DURABLE_OBJECTS_AUDIT.md for the full findings list and resolution table.
The fragile line-suffix regex this replaces was added to Peerly's own
worker/index.test.mjs by a concurrent commit ("fix: configure Peerly
preview networking") after the HeyHubs-side audit fix for the same issue
was already written; apply the identical char-scanning fix here too.
Also regenerate worker-configuration.preview.d.ts: a concurrent commit
added APP_ID to wrangler.preview.jsonc after this file was first generated.
establishSession clears the cached capability to '' on a 401 so the next connect cycle re-enrolls, but ensureCapability's `typeof cached === 'string'` check treated that empty-string marker as a valid capability and kept resending it. The server 400s an empty capability rather than 401ing it, so once a device hit one 401 it looped on /api/network/session 400 forever with no way back to enroll.
…to ws-relay durable-objects was already a fully-wired fourth VITE_SIGNALING strategy (enrollment, session, TURN, and WebRTC signaling all via Durable Objects) but was undocumented outside the internal architecture/implementation plans, and the architecture doc's "proposed implementation plan" status line was stale now that it's built and running in preview. - README.md, packages/core/README.md, .env.example: add durable-objects alongside nostr/ws-relay/supabase, marked preview-only pending the production cutover. - docs/DURABLE_OBJECTS_ARCHITECTURE.md: correct the stale status line. - docs/relay-deployment.md, docs/production-rollout.md: cross-link the newer path from the docs describing today's production relay ops. - docs/RELAY_VS_DURABLE_OBJECTS.md: new side-by-side comparison of the two coordination backends — how each works, and the concrete benefits and trade-offs of the Durable Objects approach versus the relay stack production still runs.
…oint useTurnCapability's connectivity diagnostic always resolved TURN credentials through resolveIceServers, which hits the legacy /api/network/credentials relay-ticket endpoint. Under COORDINATION_BACKEND=durable-objects that endpoint's relay-ticket secrets are intentionally absent (the whole point of the migration is to retire the self-hosted relay), so the probe got a guaranteed 503 and reported "TURN unreachable" even when TURN worked fine through the DO session. Route the probe's credential resolution through resolveProbeIceServers: on the durable-objects strategy it reads the TURN credentials the authenticated control session already returns from /api/network/session; every other strategy keeps using the legacy endpoint. The realtime runtime is imported lazily so nostr/ws-relay/supabase-only consumers don't bundle the control client.
…lly signs handleSession verified the device signature over deviceProofBytes(..., nonce, claims.sid), but the browser client (src/realtime/client.ts establishSession) signs over (purpose, app, deviceKeyId, timestamp, nonce) with no sid — it cannot bind to a sid it never receives, since the sid lives only inside the opaque, MAC-protected capability. Every real /api/network/ session request therefore failed device-signature verification and returned 401, leaving the client in an endless enroll -> session(401) -> re-enroll reconnect loop. (The earlier empty-capability bug produced a 400 loop that masked this; fixing that unmasked the 401.) The sid is already bound by verifyCapability above, so dropping it from the device proof loses no security. Add a happy-path worker test that drives the full capability + real-ECDSA-signature path handleSession expects and asserts a 200 + Set-Cookie — the origin/size tests never exercised signature verification, which is how the mismatch shipped.
registerSession hard-rejected a 4th distinct device key with cap-exceeded (409). Sessions carry a 30-day TTL and clearing browser storage regenerates the device key, so a returning user who cleared data or rotated browsers a few times would be locked out of their own account until the stalest enrollment expired — an endless enroll(409) reconnect loop with no recourse. Apply the standard "max N devices, newest wins" rule: a new authenticated enrollment that would exceed the cap evicts the least-recently-enrolled device (and closes its live sockets) rather than rejecting itself. Eviction is a capacity bound, not revocation, so the device epoch is untouched and the bumped device just re-enrolls on its next visit.
lobbyProfile requires a stored opaque user id (peerly-id-user-id), but the only backfill path ran when a workspace session was present. A signed-in user with no workspace whose user id went missing (e.g. localStorage cleared while a token lingered in sessionStorage) got a null lobbyProfile, so Home fell through to the create-workspace screen instead of the friends/DM view. Derive the user id from the live token's already-verified iss/sub claims on bootstrap when it is missing, matching what sign-in stores.
… naming Three fixes to the Durable Objects discovery path used by HeyHubs: - coordination.ts setRoom/clearRoom never caught the publishRoom/deleteRoom rejection, so a re-announce that lost the monotonic-revision race surfaced as an unhandled "conflict" promise rejection in the console. Catch it, and issue strictly-increasing revisions (Date.now() alone repeats or inverts under rapid re-announce). - UserGatewayDO never wrote an online-presence lease, so the presence-stats shard's online count stayed 0. Publish a coarse lease on connect, renew at half-life via the existing alarm, and expire it on the last clean close. A no-op for apps that do not bind PRESENCE_STATS (Peerly). - Pass the interest name explicitly into InterestQueueDO enqueue/dequeue instead of relying on ctx.id.name, which is not a dependable source inside the object across runtimes.
UserGatewayDO derived both its app and its opaque account id by parsing
`ctx.id.name`. That property is `undefined` inside the object even when the
stub came from `getByName('app:uid')` — verified against workerd — so every
gateway for every account ran as app='app', uid=''. One line, four visible
failures:
- Signal scope route ids were derived under app 'app' and authorized at
`app:<routeId>`, while the router opens `<app>:<routeId>`. Every signal
socket got 403, so no two peers ever exchanged SDP: room joins showed each
participant alone, and both then correctly elected themselves host.
- InterestQueueDO.seeks is keyed by uid, so each new seeker OVERWROTE the
previous one. Two people picking the same interest could never be two rows:
no match, and an availability count stuck at one.
- Invites and DM rings were delivered to a phantom `app:<uid>` gateway that
nobody has a socket on.
- The presence lease was suppressed (empty uid) and written to shard `app:0`
while /api/stats/snapshot reads `<app>:0`, so the online count stayed 0.
The app now comes from `env.APP_ID` — what WorkspaceDO and InterestQueueDO
already do — and the uid is handed in by each authenticated entry point
(enroll, session validation, control-socket upgrade, cross-DO delivery) and
persisted in `meta`. Existing objects self-heal on the next session request.
Two further causes of "could not connect to peer after exchanging SDP":
- The TURN REST credential is minted with cookieTtlMs (10 minutes) and the
client cached it in memory for the life of the tab. Past ten minutes peers
offered each other credentials coturn had already expired, and new signal
sockets 401'd on the expired pnet cookie. RealtimeClient now re-runs
/api/network/session every 4 minutes, refreshing both, and re-dispatches
`turn`. Adds a WebSocket keepalive ping (answered by the existing
auto-response, without waking the object).
- DO builds handed Trystero a TURN-only ICE list where the legacy path sends
STUN + TURN, and joinRoom replaces Trystero's defaults with it — so those
builds had no reflexive-candidate source at all. Now identical on both
backends, with the documented transport ladder from docs/turn-relay-vps.md.
Also: seek exclusions are carried in the app's own id space (`memberId`)
instead of being compared against the server's opaque uid, which no client
can name — a blocked person was still matchable; `getDurableObjectsIceServers`
waits (bounded) for credentials instead of silently reporting no TURN when a
connect cycle only scheduled a retry; clearing a seek no longer stops the
interest-count poll or drops an incoming match.commit; directory.publish
reports cap-exceeded/too-large instead of flattening them to conflict; and
`<label>-preview` staging hostnames are accepted alongside
`<label>.preview` (a worker at dev-preview.peerly.cc had every
/api/network/* request rejected 403).
Regression tests cover the identity plumbing; the HeyHubs side adds an
end-to-end seek.start test that fails against the old behaviour.
A behavioural pass over the control plane, after the identity fix in fc3f168 showed the previous audit had only ever checked structure. Findings and the process change are recorded in docs/DURABLE_OBJECTS_AUDIT.md. Security — device revocation did not revoke anything server-side. UserGatewayDO.revokeDevice had no caller at all. Revoking a device in Peerly's UI ran a purely local function that dropped a peer-to-peer grant, while the revoked device kept a valid 30-day capability, its server session and its open control socket. Adds the `device.revoke` command (the account is the socket's own identity, never a parameter, so it can only reach the caller's own devices), emits `device.revoked` to the account's other devices, exposes `revokeRealtimeDevice()` from core, and wires the devices page to it — local first so it still works offline, then the control plane, surfacing a failure rather than silently doing nothing. Reliability — the resume cursor lived only in memory, so every page load sent resumeSeq 0, which the gateway reads as "start of the retained stream": each reload replayed up to 24h of invites, rings and match commits. Now persisted (debounced, flushed on disconnect), which also makes the aged-out-cursor snapshot path reachable for the first time. Efficiency — SignalScopeDO never used its own routing: every SDP/ICE frame was broadcast to the whole scope, so signaling was O(N^2) in a room and every participant received every other pair's envelopes. Participants now claim the topics they listen on and delivery is routed; an unclaimed topic still broadcasts, which is what keeps Trystero's room-wide announce working. Only the envelope's topic strings are read — the message beside them stays opaque. Efficiency — polling was neither demand-driven nor visibility-gated, contrary to the architecture plan's own constraint. `directory.list` bills a cross-object request on the shard per tab per poll: at 10s a single idle background tab spent ~8,600 Durable Object requests a day, about a tenth of the whole free daily allowance. Stats stay at 10s (an edge-cached GET, and the number a user actually watches); rooms move to 30s; both stop while the tab is hidden and catch up on becoming visible. Also: `invite.ack` drops the mailbox copy instead of leaving the cap to evict unread invites, and `scope.leave` releases the authorization (closing that device's socket) instead of letting it live out its lease. WorkspaceDO is recorded as a decision rather than fixed: it is unreachable (no workspace.* command, no route, no non-test reference to env.WORKSPACES) and stays that way — the Phase 5 cutover config should omit it rather than inherit a permanently orphaned class, since production has no tag history yet.
Reverts the transport-ladder expansion I added while chasing the "could not connect after exchanging SDP" failures. `expandTurnUrls` rewrites `turns:<host>:5349` to `turns:<host>:443`, which only reaches coturn where nginx SNI-routes 443 to it — docs/turn-relay-vps.md configures that map for turn.peerly.cc and not for every TURN hostname. So the expansion did not add a fallback, it *replaced* a configured, working TLS endpoint with one that may terminate somewhere else entirely, and on a UDP-blocked network that is the difference between connecting and not. The legacy relay path this backend has to match sends TURN_URLS exactly as configured. Differing from the known-good path while diagnosing a regression against it was the wrong move. The STUN entry stays — that part genuinely did restore parity. Also refresh the interest counts as soon as a seek publishes, so the person who picked an interest sees their own count move immediately instead of waiting out the poll interval.
First slice of the full rewrite (docs/REWRITE_ARCHITECTURE.md). Additive: the existing realtime modules keep serving until each replacement passes the same tests, so nothing deployed changes yet. `packages/core/src/protocol/` is one pure, platform-free source of truth for the wire contract, imported unchanged by both the browser client and the Worker. The previous split — limits.mjs plus a hand-copied limits.ts whose own comment admitted "keep in sync by hand", and two frame codecs — put security-relevant constants and the envelope contract in two places and relied on a drift test to notice. That was a build-configuration workaround, not a design (audit C1/C2). Audit findings designed out rather than patched: - G1: frame ids come from an IdSource seeded with per-client entropy. The old `Date.now()+counter` format collided between two devices of one account that loaded together, and the second silently received the first's cached ack from the gateway's idempotency table. Test asserts 1000 ids across two sources never intersect. - G3: an empty seek has no representation. normalizeInterests returns null rather than an empty array, so the caller cannot write a seek row that enqueues into nothing and leaves the user matching nobody forever. - Blocklist id-space: MemberId and OpaqueUserId are distinct branded types, so comparing an exclusion list against the server's opaque id — the bug that silently disabled every blocklist — no longer compiles. - G6: directoryMaxRoomsPerOwner added, so one publisher cannot fill a shard and lock every other account out. - Version mismatch closes 4002, keeping the client's terminal upgrade path reachable; the bye frame the guide specified now exists. The app boundary is structural, not advisory. Core declares only universal commands (hello, resume, scope.*, invite.*, ring.send, device.revoke) and an app extends the registry with its own — so matchmaking and room-directory commands stop living in core and being deployed inert in the app that has no use for them (audit A1/A2). Registries are immutable; extend() returns a new one and refuses to shadow an existing type. boundaries.test.ts enforces the dependency rule instead of documenting it: protocol imports nothing outside its layer, touches no platform API, and names no consuming application. It caught a violation in this very commit — a module comment that named an app — which is the point. 40 new tests. Full suite green: 453 unit, 71 workers.
added 17 commits
July 28, 2026 09:57
Four things from a real testing session. A pasted invite link did nothing. A link differing from the current one only after the `#` does not reload the document and never fires `popstate`, so nothing re-read the hash — the invite was sitting in the address bar being ignored. `useBrowserHistory` now listens for `hashchange` too, and its callback is `onLocationChange` because that is what it means. The open conversation was component state, so a refresh dropped the user back to the list. It is now part of the route: `/friends/<userId>`, with the list at `/friends`. That also answers the other half of the report — the URL said `/home` for what the app calls Friends everywhere else. `/home` still parses, so existing links keep working. A forgotten workspace stayed in the rail until something unrelated repainted, which looked like the deletion had failed. The rail was read during render, and a localStorage write tells React nothing. The store now announces changes and the rail holds state. Not fixed here, and worth stating: an invited user still gets no notification of a workspace invitation. That is a missing feature rather than a broken one — invites travel by link today — so it needs its own change.
Landing on /friends/<id> showed the friend list and opened the conversation some seconds later. The friend was known straight away — that list loads synchronously — but the room code was computed inside the click handler, which a refresh never runs. So the pane had a friend and no code, fell back to the list, and only opened when a later render happened to fix it. The code now derives from whichever conversation the route names, so arriving by click and arriving by refresh take the same path.
Two halves of one problem, at your call on the trade-off. The ID token now lives in localStorage. It was session-scoped on the reasoning that a bearer credential should not survive a restart; that bought little and cost a sign-in every single time, because the token expires hourly anyway and every peer re-verifies it rather than trusting our copy. What bounds the exposure is that expiry, not where the string sat. Old sessionStorage tokens are migrated on first read, so an open tab is not signed out by the upgrade itself. Storage alone would only have moved the interruption from restart to the next hour, so `useIdentityRenewal` keeps a live token without asking. It fixes two faults: the re-auth prompt was mounted inside the workspace component, so a token could expire on the friends or DM screen with nothing offering to renew it — the app simply stopped working — and it reacted to expiry instead of anticipating it, so requests failed first and the user was told second. Renewal now runs app-wide, five minutes ahead, and on returning to a tab that slept through its own timer. `requestGoogleCredentialSilently` resolves null rather than throwing when Google declines — several accounts, consent withdrawn, third-party cookies blocked. That is a normal answer, and a failed silent renewal must never turn into a popup nobody asked for; the visible sign-in path still handles those. The E2E test asserting the token was *not* in localStorage encoded the old posture. It now checks what still has to hold: the token stays out of the persisted session record, and — pinned in deviceSync.test.ts — out of device sync. The closed allow-list is the only reason this change did not also start copying a bearer credential to every paired device. Not verifiable here: the silent path needs real Google, which the E2E bypass replaces. It ships to preview unconfirmed.
This reverts commit 19e2cdf.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 12 commits
July 29, 2026 10:13
Callers await connect() before every command, so an app with polling timers calls it many times a second. A failed cycle scheduled its own retry and then resolved, which left connect() free to start another cycle beside the pending one -- and each of those scheduled another, because scheduleReconnect() overwrote the timer handle rather than clearing it, and an overwritten handle still fires. One transient 500 grew into ~400k enrol/session requests in three hours and exhausted a preview account's daily Durable Objects quota while nobody was using the app. Zone analytics for the burst: 59,798 successful /api/network/session calls in a single hour against 971 WebSocket upgrades. - connect() returns early while a retry is armed; queued commands wait on their own deadlines instead of dialling again - one beginCycle() so the retry timer and every connect() caller share a single in-flight attempt - scheduleReconnect() clears the armed timer before arming another - equal jitter instead of full jitter, which had no floor - backoff resets only after a connection outlives a stability window, not at the open handshake -- a socket the server accepted and closed immediately used to put the next attempt back at the 250ms base
prune() ran `seq NOT IN (SELECT seq ... ORDER BY seq DESC LIMIT ?)`, testing every candidate row against a 1000-row list with no index to find the candidates with, so one prune cost roughly rows x keepNewest reads. It runs on every alarm; the account showing the reconnect storm had read 300M SQL rows against 521k written. Replaced with a single non-correlated lookup down the seq primary key for the first row outside the retained window, plus an index on created_at to bound what the delete visits at all. Same semantics -- the shared GatewayStorage contract covers it for both adapters.
batchWindowMs, batchMaxEvents and batchMaxBytes were declared, and addToBatch/shouldFlush implemented, but emit() never called either -- only a domain unit test did, so every event was still its own send. emit() now coalesces into one delta frame per batchWindowMs under the item and byte caps. The append stays immediate: the stream is the durable record a resuming client is owed, and only the send is delayed. - one pending flush via a new Scheduler port, never setInterval, as DURABLE_OBJECTS_IMPLEMENTATION.md 6.6 requires. Not an alarm: an alarm is a billed request, and one per 75ms batch would cost more than the sends it saves - urgent kinds go out immediately, after flushing anything already pending so deltas cannot arrive out of sequence. Core keeps device.revoked; an app adds its own through app.urgentKinds rather than core learning about them - flush before accepting a socket, so an arriving client cannot both receive a batch opened before it existed and replay it from its own resume cursor - app.migrate hook, run after app.schema on every construction, for objects whose tables predate a column
The 2026-07-24 structural audit asked whether the boundary between the two apps was clean. It now is. The 2026-07-23 closeouts reconciled a security audit that has since been actioned. Both are removed. What replaces them asks the question the 2026-08-02 incident exposed: what the control plane costs, and who can make it cost more. A preview deployment nobody was using exhausted the account's daily Durable Objects quota, and the notable number was not the 400k failed requests but the 59,798 *successful* session establishments in one hour against 971 WebSocket upgrades. Nine findings, all with file:line evidence, a measured per-tab cost model, and a resolution table. Citations are repointed rather than left dangling: eleven places referenced the removed files, six of them naming "finding A1" from a code comment. Those comments already state the reasoning, so they keep it and lose the reference.
Remediates the nine findings in CONTROL_PLANE_AUDIT_2026-08-02.md. They land together because they share `limits.ts`, and because they are one idea: nothing stood between a misbehaving client and the account quota. C1 — `/api/network/enroll` and `/api/network/session` had no rate limit of any kind, and each performed two Durable Object RPCs before any throttle could apply. `AUTH_RATE_LIMITER` now caps both, keyed by device key with an IP fallback. It stays open when unbound: this is a cost control, not an authorization one. C2 — a gateway failure propagated out of `fetch` as an unhandled 500, which every client reads as retryable, so the symptom of an overloaded control plane was a stampede against it. Failures are caught at the route boundary and answered 503 with `Retry-After`, and the client uses that as a floor on its next backoff. C3 — `invite.send`/`ring.send` named an arbitrary recipient with no relationship check, and a full mailbox evicted its oldest entry. Every entry present is unread by construction, so that was a way to clear a stranger's mailbox at five entries a second, from any account that had learned their opaque id. Now: a per-recipient token bucket, and a full mailbox refuses rather than evicting. The sender is told `cap-exceeded` instead of being told it was delivered. C4 — the room directory was polled every 30s at two DO requests each, the largest per-tab cost in the system. `directory.watch` takes a lease instead; a lease and not a subscription because a shard cannot observe a browser tab closing. C5 — one constant was both the socket ceiling and the device ceiling, so raising either raised the other. Split; devices go to 4, which is the ordinary shape of one person's hardware. C6 — `accept()` closed one socket over the cap even when the object held many, and hibernated sockets from closed tabs count until reaped. C7/C8 — `cookieTtlMs` 10 -> 30 minutes, and the paired auth RPCs become one call each. Together these take an idle tab from ~960 DO requests a day to ~240. Revocation is unaffected: a socket is admitted against the session epoch in storage, not the cookie's lifetime. C9 — `mailbox_created` index; coordinator availability follows the transport's `state` event rather than `connect()` resolving, which reported an available coordinator on a transport that never connected.
`worker:check` compares the generated types against the config and fails when they drift. Adding the rate limiter binding without rerunning `wrangler types` is exactly the drift it exists to catch.
A device key is a keypair the client generates. Limiting the enrol and session endpoints on it alone hands a fresh allowance to anyone willing to call generateKey again, which is not a limit — so AUTH_IP_RATE_LIMITER now runs alongside it and both must allow. Writing the tests found a worse case than the one I set out to fix: a request carrying neither a device key nor an address skipped *both* buckets and went entirely uncounted. Requests without an address now share one bucket rather than bypassing the check, since Cloudflare always sets that header and a request without it did not arrive the usual way. The limiter also says when it is not working. It fails open by design — this is a cost control, not an authorization one — but a silent fail-open is indistinguishable from a working limiter, which is how you learn it was broken from a quota email rather than from your own logs. Separately: `signalSocketsPerDevice` and `maxTokenStringBytes` are gone. Neither had a single reference in either repository. `limits.ts` claimed the first was a *past* finding, already dealt with; it never had been. A per-device signal socket cap is a feature to build, not a number to declare, and `maxTokenStringBytes` was redundant beside the request body cap that already bounds the same input.
`limits.ts` has said since the rewrite that adding a limit is not enough, because `limits.enforcement.test.ts` fails for any key nothing enforces. That file did not exist. It is why the delta-batching trio shipped declared and unread for a release, and why two further caps were still dead when I went looking. Every key must name the file that enforces it, and that file must actually read it — a registry nobody verifies is a comment. Rate-limit ceilings are enforced by a Cloudflare binding rather than by code, so those are checked against the number the wrangler config declares, which catches the constant and the deployment drifting apart. The last block is the specific failure this exists to catch: a constant read only by a function nothing calls. `addToBatch` and `shouldFlush` were implemented, unit-tested, and reachable from no production path — enforced from every angle except the one that mattered. It found four stale claims in its own first draft, and has since caught config drift twice while building vendored tarballs.
On 2026-08-02 a client bug spent 4.8x the daily Worker request allowance, 2.6x the Durable Object request allowance and 60x the rows-read allowance between 07:00 and 09:00 UTC. The first anyone knew was a quota email the next day, and the rows-read one arrived later still. Every fix in that incident was cheap; the twelve hours of not knowing were not. A cron Worker reads the account's own analytics every fifteen minutes and reports at 25/50/80/100% of each budget. Fifteen rather than sixty because the failure mode is a runaway loop that covers a whole daily budget inside an hour — an hourly check would routinely report a budget that was already spent. Its own two failure modes get the most care: - It throws rather than reporting zero when analytics fails. A watcher that silently reports zero is indistinguishable from a quiet account, which is exactly the state it exists to break. - It reports only on a *rise* in level, so a budget parked at 60% alerts once instead of ninety-six times a day. An alert you mute is worse than no alert, because it reads as coverage. A separate Worker rather than a handler bolted onto an app: the budgets are account-wide, so one watcher covers both, and a fault here cannot take a user-facing route down with it. No Durable Objects — a watcher that spent the budget it watches would be a poor joke. Needs CF_ANALYTICS_TOKEN (Account Analytics: Read, nothing else) and CF_ACCOUNT_ID before it can do anything; it says so loudly rather than running quiet. USAGE_ALERT_WEBHOOK and a USAGE_STATE KV namespace are both optional, and their absence degrades toward more noise rather than less, which is the right way round for an alerting path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
UserGatewayDO,SignalScopeDO,WorkspaceDO, plus the HeyHubs-onlyInterestQueueDO/PresenceStatsShardDO/RoomDirectoryShardDO(implemented in@peerly/coreso both apps share one codebase and test suite).COORDINATION_BACKEND=legacy-relay; no existing signaling/workspace code path is touched (Phase 1 exit criteria).wrangler.preview.jsoncthat deploys the DO-enabled staging worker. It cannot live inwrangler.jsonc: Workers Builds deploys every non-production branch withwrangler versions upload, and Cloudflare rejects versions carrying a DO migration (API error 10211). Migrations enter the default config only in the Phase 5 cutover PR, where merge-to-main applies them via a full deploy.Where to test what
COORDINATION_BACKEND=durable-objects. Deploy withnpx wrangler deploy -c wrangler.preview.jsonc.legacy-relay); realtime routes intentionally answer 503 here. Google sign-in cannot work on per-branch hostnames (exact-origin OAuth), and DO namespaces cannot be created by version uploads — so the DO path is untestable on branch aliases by platform constraint, not by omission.Test plan
npm test— 398 unit tests passnpm run test:workers— 45 Durable Object tests pass (protocol boundaries, token rotation, device signatures, gateway hibernation/resume/idempotency/match reservation, scope isolation, workspace presence fan-out)npm run lint,npm run worker:check(now dry-runs both configs),npm run buildpasswrangler versions uploadverified locally — the exact command Workers Builds runs for branches — succeeds with the migration-free default config