browser peer + playground: fluree-db-browser, @fluree/db-wasm, SSE head tracking, and the residency retry loop

[aaj3f]

This is the second and final PR of the wasm program: everything that turns PR-1's "the engine runs on wasm32" into product. It's deliberately over-scoped as one reviewable stack (per our fewer-bigger-PRs preference) — 13 commits in four layers, each layer gated independently before assembly. Stacked on wasm/groundwork; nothing here touches the native engine paths beyond what's called out below.

What you can actually do at this HEAD:

• Run Fluree in a browser tab, no server, no special headers. cd fluree-db-wasm/js && npm install && node scripts/build.mjs && node scripts/serve.mjs → an interactive playground at 127.0.0.1:8787/demo/ (create ledger → JSON-LD insert/upsert/update → SPARQL + JSON-LD queries, plus a transfer-vs-clone transport timer). Headless proof: node scripts/smoke-browser.mjs — the CI-shaped smoke drives the shipped npm package over CDP and passes with init ~900 ms, insert ~740 ms, query ~675 ms on the dev box.
• Assemble a browser cache peer against a real server. fluree-db-browser::connect(api_base, token, config) yields a BrowserPeer over the peer-proxy wire contract: CID-verified fetches through a channel-bridged transport, an IndexedDB CAS cache with verify-once-then-trust and LRU byte budget, a synchronous residency tier feeding the engine's read path, SSE head tracking, and the production retry loop that makes cold queries converge. (The JS shell's peer mode — wiring BrowserPeer into the worker protocol — is the one deliberately-deferred integration; the constraints are enumerated in fluree-db-wasm/js's protocol notes and both sides were built to slot together.)

The four layers

1. Server browser-readiness + the transport seam (fluree-db-server, fluree-db-nameservice-sync). CORS now uses AllowHeaders::mirror_request() — the first cut hand-listed allowed headers and our internal review caught that it would have regressed every browser client that sends fluree-identity/fluree-track-*/Idempotency-Key (the wildcard it replaced doesn't cover Authorization, which is the whole reason to touch this; mirroring covers both). Object GETs get ETag: "{cid}" + immutable caching (CID-addressed bytes are the textbook case — this also slots the browser's HTTP cache underneath IndexedDB for free); the NS record gets an ETag over the served bytes (a digest, not the {commit_t}:{index_t} composite we first tried — retracted/branches/serving all change without a commit, which review also caught) with 304 handling ordered after every authz/serving gate so a 304 can never become an existence oracle (test-pinned, including out-of-scope-token → 404 with a matching validator). The proxy clients now sit on an HttpTransport seam (plain-data request/response, Bytes bodies, no reqwest types) so a wasm fetch impl can slot in; native behavior is preserved to the error-message level (all four call sites diffed line-by-line in review), and ReqwestTransport + the native machinery are target-gated so fluree-db-nameservice-sync genuinely builds for wasm32 — which CI now checks.

2. The residency retry loop + freshness (fluree-db-api, fluree-db-binary-index, fluree-db-core). The production form of PR-1's machinery: query_with_options on residency builds (wasm32, or the residency dev feature natively) wraps execution in drain-wants → concurrent fetch-and-pin → re-run, holding the storage's query guard across rounds (a new default trait method mirroring miss_register; the guard is what makes progress monotone — eviction is frozen while any query is in flight) with a fresh fuel tracker per round so retries don't bill earlier rounds against the final budget. Scans rarely reach this loop at all: the operator-frame retry inside binary_scan absorbs leaf misses in place (mutation-tested — disabling the outer loop only fails the one-shot fast-path and policy tests, exactly the intended division of labor). Freshness keeps novelty replay (no new consistency mode) and closes its miss source by prefetching the overlay's reverse-dict leaves at ledger load, pinned by a mutation-checked multi-leaf unit test. Streaming query entry is excluded on wasm by design (rows emit before completion; can't re-run). Not yet wrapped: the cypher/tracked/dataset entries — named follow-ups, same pattern.

3. fluree-db-browser — the engine-side browser I/O crate. A Send + Sync channel bridge to a spawn_local driver that owns fetch/AbortController/IndexedDB (the storage traits box Send futures, so a JsFuture-holding transport can't exist — this is the shape that satisfies them); WasmFetchTransport implementing the seam to its documented contract (Ok for any HTTP status — status semantics stay client logic; credentials: 'omit'; exactly one JS→wasm copy per block, documented); the residency tier (Arc-clone hits, pin sets, batch-first coalesced fetches, no-eviction-while-in-flight with a bounded deferred-insert wait); one memory governor deriving every budget from a single ceiling; write-behind IndexedDB persistence with fetch-admission backpressure (a lagging IDB stalls the pipeline instead of queueing clones). The integration test to start review at runs the engine's landed RetryBudget::after_error through the real content_store_for bridge over this storage — miss → register → drain → fetch → pin → re-run with one network fetch, and empty-register → "real error" discrimination — every step positively asserted. 4 real-browser tests (headless Chrome) cover fetch round-trips, error classification, and IndexedDB persistence across reopen.

4. fluree-db-wasm + @fluree/db-wasm (name pending the npm-org decision) — the JS shell. The engine in a dedicated worker behind a typed protocol; a main-thread proxy with hand-written TS types; per-call frozen snapshot handles (an update mid-session provably doesn't leak into an open snapshot — test-pinned) and no streaming surface; a crash model where a wasm trap marks the worker fatal, in-flight calls reject with typed engine_crashed/out_of_memory, and the proxy recycles the worker so the Playground object survives; maxMemoryBytes (default ¼ of deviceMemory, clamped) applied per query. SSE head tracking rides a shared pump lifted from the server's peer subscription loop (parameterized over connection/timer/sink; the server keeps its public shape and drives the same pump), consumed browser-side as a dedicated streaming driver job — with a real ReadableStream test in Chrome. Ships with a wasm-smoke CI job that executes the browser build (bindgen tests in a dedicated worker + the shipped package over CDP), not just compiles it.

Performance and build characteristics, transparently

Native first, because it's the thing we rank highest. The shared-crate surface of this PR is deliberately tiny: the server header/CORS commit, the HttpTransport seam under the proxy clients (reviewed line-by-line for behavior preservation down to error-message text; per HTTP round trip it adds one async_trait box and a small header Vec — invisible next to the network — and the response body stays Bytes uncopied), the SSE-pump lift (server keeps its public shape and drives the same pump), and two default trait methods on the storage traits (query_guard/miss_register, default None/None — native implementors never override them and the retry loop that consults them doesn't compile into default native builds). Everything else is two brand-new crates. The full native battery is green at this HEAD — workspace check, nameservice-sync 96, proxy_integration 34 (serial), browser crate 41, residency suites 5/5 + 380, query lib 1416, fmt/clippy per crate — and since this touches perf-relevant paths the bench-compare gate will annotate TIME and PEAK_MEM on the PR; treat that as part of the review.

Wasm runtime characteristics, measured not guessed (dev box, headless Chrome, shipped package over CDP, no isolation headers): engine init ~901 ms (includes wasm streaming-compile + worker spawn + first instantiation — a one-time cost per page), JSON-LD insert ~739 ms, SPARQL query ~675 ms on a fresh micro-ledger. Result transfer between worker and main thread uses transferred ArrayBuffers rather than structured clone where size warrants it — the measured gap that motivated this is ~300 ms vs ~6.6 ms at 32 MB. On the peer read path: a synchronous residency hit is an Arc clone (zero copy); a fetched block is copied exactly twice (JS→Bytes, Bytes→Arc<[u8]> — the seam contract's price, documented in cas.rs), and the Vec-returning legacy Storage methods pay a third at their boundary, which the residency-first entry points avoid. IndexedDB writes are write-behind and can never delay a read; a lagging IDB applies backpressure to fetch admission instead of queueing clones.

What a cold open actually costs, measured (not in this PR to fix, but reviewers should have the number): opening a 200-row ledger in a browser peer issues 18 object GETs — and on a 100 ms link they arrive as 16 sequential waves, 15 of them a single object, because the peer discovers what it needs a round at a time. Roughly 37 serialized round trips to move 20 KB. Two consequences worth knowing while reading the CORS commit: Authorization makes every object GET non-simple, so each one costs an OPTIONS first (~1.9 s of a ~5.8 s cold open at 100 ms RTT), and the Access-Control-Max-Age we set genuinely helps the repeated-URL routes but buys nothing on the object route, because preflight caching is keyed per URL and every CID has its own URL. Also measured: Range is CORS-safelisted (so the ranged leaf reads planned later add no preflight of their own), and a stable batch URL preflights exactly once ever. None of this is urgent — objects are immutable-cached, so it is paid once per cold profile and never again — but it says where the next work goes: a batch object endpoint (one preflight, ~4× fewer requests, drops into the driver's already-batch-shaped fetch_cids) and widening what each discovery round learns. The peer is discovery-limited, not width-limited: max_concurrent_fetches defaults to 8 and currently sits unused because the engine never knows about 8 objects at once.

Build/size: shipped wasm (fat LTO, -Os, wasm-opt -Os): 8.99 MB raw / 3.31 gzip / 2.28 brotli, 26 KB JS glue. A pure-size build (-Oz, one env flip documented in build.mjs) lands at 7.48 / 2.83 / 2.01. We shipped -Os on the speed-first mandate — -Oz trades speed-relevant codegen for ~12% wire size and I'd rather hold the line on speed until someone shows the download matters; flip is one env var if the team disagrees. For calibration: this binary still links r2rml, the reasoner, BM25, and Cypher — feature-slicing a slim profile is available headroom we deliberately haven't spent (DuckDB-wasm ships ~11 MB compressed; we're at 2.3 with everything in).

Decisions that warrant scrutiny (and the alternatives they beat)

• CORS via AllowHeaders::mirror_request(), not an allow-list. Our first cut hand-listed headers and internal review proved it would regress every documented fluree-*/Idempotency-Key browser caller (the wildcard it replaced doesn't cover Authorization — the reason to touch this at all — but covered everything else). With nothing to keep in sync. It is not, however, "exactly the wildcard's trust posture" — an earlier draft said that and review correctly pushed back, because the difference is the point of the change: * deliberately excludes Authorization, so cross-origin authenticated requests were impossible before and are possible now. This is strictly more permissive, on purpose. The consequence is real for internal deployments — a page the user visits can reach an internal Fluree server with an Authorization header and read the response. That is not CSRF (no ambient credentials; the page must already hold a token), but it widens the browser-as-pivot surface. The code comment now says this rather than the softer version. Follow-up, deliberately not smuggled in here: cors_enabled is a bool with allow_origin(Any) hardcoded, and now that browser clients are a first-class use case an origin allow-list is worth having.
• NS ETag = digest of the served bytes, not the {commit_t}:{index_t} composite we tried first — review found four ways the representation changes without a commit (retracted, branch count, serving, drop-recreate), each of which would have pinned a stale 304 in browsers indefinitely. Bytes-digest is strictly stronger than any field list.
• Eviction policy: frozen-while-any-query-in-flight, newcomer waits on overflow. Alternative: per-query pin attribution. Rejected because pins can't be attributed through a resolve_cached_bytes(&cid) hit — a query that hits bytes another query fetched would hold no pin, and its working set could vanish between retry rounds (livelock). Tier-global freeze is coarser but is what makes retry progress provably monotone. The cost: a memory-hungry query can hold the tier open; the bounded budget_wait + typed failure is the escape.
• The transport buffers whole bodies; SSE is a separate driver job. Alternative: a streaming HttpTransport. Rejected — every CAS consumer wants whole verified blocks (CID verification needs the full payload anyway), and the one genuine streaming consumer (SSE) gets a purpose-built job type instead of complicating the seam every block read rides.
• Retry loop lives at the api query entry (cfg-gated), not in the JS shell. Alternative: shell-side loop. Rejected per the test-guard lesson: api-level means a native CI test with a miss-injecting store exercises the production loop — shell-side would leave it untestable until a browser harness, and unusable by non-JS embedders.
• Worker-per-engine on the stable toolchain; no wasm threads. The +atomics build (wasm-bindgen-rayon) needs a pinned nightly, a second artifact, and pushes COOP/COEP isolation headers onto every embedding page (breaks OAuth popups; no Safari credentialless). Deferred behind a profiling trigger, not ideology — the ladder (OPFS sync reads → multi-worker → threads) is documented and each rung is additive.
• -Os over -Oz — argued in the build section above.

Two deliberate server behavior deltas (flagged for review, not buried)

The SSE lift carries two argued changes: head-change notifications now populate source_branch/branches from the wire (previously hardcoded None/0 — branch lineage the peer client provably needs), and an unparseable ledger_id in one event degrades to using it verbatim as the name instead of erroring the whole stream. Both are in the lift commit's body with rationale; neither has a behavioral test upstream because — honest note — no pre-existing integration test exercises PeerSubscriptionTask at all. The pump got six unit tests (parse/backoff-growth-and-reset/failure/malformed-frame/stop) in the lift, but the peer-subscription coverage gap predates this PR and is worth its own issue.

What this PR does not do

No peer mode in the JS shell yet (the BrowserPeer ↔ worker-protocol wiring is enumerated and deferred — the protocol needs an unsolicited-event channel first, and a recycled worker's replayed init must never carry a raw token, so the token flow needs its callback bridge). No cypher/tracked/dataset retry wrapping. No OPFS cache tier, no ranged/pack-paged fetches, no wasm threads — all deliberately Phase-3, gated on measurements. The public/anonymous server tier stays its own future PR before any hosted playground.

Review map: start with fluree-db-browser/src/cas.rs (the storage + the after_error integration test), then fluree-db-api/src/view/query.rs (the loop), then the SSE lift (fluree-db-nameservice-sync/src/head_stream.rs against the old peer/subscription.rs), then the shell's protocol.ts/worker.ts. The server header commit is small and its tests tell the story. The Cargo diffs are wide but mechanical.

Pre-review remediation (before undrafting)

Before asking anyone to spend review time here, we ran this PR through an internal blind pre-review (a reviewer with no context on the work, applying the repo's own review rubric, checking out the PR like any colleague's). Verdict was request-changes (one CI-gate red, six highs) — all resolved. Every finding was then either fixed or rebutted with evidence — the remediation commits are on the branch, findings referenced in their messages. Highlights: a clippy lint only reachable under --all-features (invisible while stacked — the base branch gets no CI) fixed; the residency tier's eviction now uses epoch-tick semantics (entries older than the oldest live query guard are evictable), closing a warm-full-tier scenario where every cold fetch would stall then fail — with the monotone-progress property re-proven by tests at tier, storage, and api-loop levels; write-behind backpressure actually applies now (permit acquired inside the fetch slot, buffer_unordered batches) with a stalled-persist throttling test; fluree-db-browser's five real-browser suites now run in CI (they previously never executed anywhere); JS snapshot handles are generation-stamped so pre-crash objects reject typed instead of silently reading a recycled engine's state, respawns are capped with backoff, and the worker-spawn pattern is the bundler-statically-detectable shape (Vite verified by a repro script; other bundlers honestly marked unverified). One engine fact surfaced while making the memory-budget browser test deterministic, worth its own follow-up: the per-query memory budget charges grouped-aggregation state (GROUP_EST_BYTES) and hash-join lanes, but plain cross-join row retention and GROUP_CONCAT payload bytes are not charged today — the test pins the typed-rejection contract on a charged lane and says so.

Review remediation

Four items from review, all addressed; the two flagged as wanted-before-merge are the first and last.

Vary: Authorization on CAS object responses — a real cross-principal cache leak, now fixed. These bodies are authorized by bearer token and served private, max-age=31536000, immutable. private keeps them out of shared caches, but the browser's own cache is per profile, not per principal, and immutable means it never revalidates — so after principal A read a ledger, principal B on the same browser profile was served those CIDs directly, with the server never seeing B's token and never getting to refuse it. Vary puts the token in the cache key. Pinned at all three object responses (200, 206, 304) in test_object_endpoint_immutable_caching_and_conditional_get, and mutation-checked: reverting the source with the assertions kept fails the test. The NS routes deliberately don't need it — no-cache revalidates, so authorization re-runs server-side before anything is served.

The two subscription behavior deltas are now pinned. Review's point stood: peer/subscription.rs is −377/+152, a restructure carrying two deliberate behavior changes, and the six new tests covered the lifted SSE pump rather than those changes — so a regression would have had no failing test and no clean bisect boundary between "the refactor broke it" and "the intended change broke it". Two tests added at the conversion site (server_sse.rs): one pins source_branch/branches being carried from the wire (they were hardcoded None/0, so a branched ledger arrived looking unbranched) and that their absence still defaults cleanly for an older server; the other pins that an unparseable ledger_id degrades to the verbatim id plus the wire's branch rather than tearing down the stream, with a precondition assertion that the fixture really is an id the canonical parser rejects, so the test cannot go vacuous. The first is mutation-checked against the hardcoded values.

The duplicated query entry path is factored into one plan_and_execute_round. Both cfg arms now call it and differ only in the retry wrapper. This one is worth stating precisely because it interacts with the dev-dependency feature unification raised on #1714: cargo test -p fluree-db-api and every api bench compile the residency arm, so the production cfg(not(...)) arm is not built there at all — two copies of the main query path meant nothing in the api suite exercised what ships, and drift between them would have been invisible. The call is once per query, not per row, so it is not on any hot path; verified building in all three configurations (production native, residency, wasm32).

Caught up to main, and where the browser-driver hardening lives

Rebased onto current main (verified semantically, not just textually). A second independent adversarial review then covered the two new crates (which the human review explicitly excluded). Its findings are fixed in the stacked child #1733, not here, because they are entangled with lq-smoke's token-cell evolution (pre-this-PR there is no TokenCell, so the driver-open/deadline/timer commits cannot be cleanly cherry-picked down without pulling a dependency chain). This is a strict stack that merges in order, so #1715 → #1733 lands the browser peer as a unit: the driver-open hang, the u32::MAX-is-immediate timer, the SSE connect timeout, the unverified-cache and token-Debug issues, and the buffer bounds are all closed by the time #1733 merges. #1715's own head is the transport/CORS/header surface the human review approved; its browser-crate hardening completes in #1733.

[bplatz]

Approving. Reviewed the native/shared surface and the server-side security posture — not the two new crates (fluree-db-browser ~4k lines, fluree-db-wasm, the JS shell), so don't read this as a full sign-off on those. Four inline items; the first and last are the ones I'd want settled before merge.

Verified clean:

• The storage trait additions are inert: query_guard/miss_register are default-None plus mechanical forwarding through Arc<dyn …>, StorageContentStore, BranchedContentStore. No native implementor overrides them; every consumer is cfg-gated.
• The query.rs non-residency arm is the original code verbatim — production planning/execution untouched.
• The NS-record conditional GET is correct: no-cache forces revalidation so authz is re-checked every time, and the 304 is ordered after every authz/serving gate. The bytes-digest ETag over the composite is the right call.
• "No pre-existing test exercises PeerSubscriptionTask" — grepped, accurate.

[bplatz]

Suggest adding Vary: Authorization to the object responses.

These are authorized by bearer token and cached private, max-age=31536000, immutable. private is the browser's own cache, shared across sessions on one profile; immutable means it never revalidates. So after principal A reads a ledger, principal B on the same browser profile gets those CIDs served from cache with the server never seeing B's token.

The comment here already reasons about the token (private "because the response was authorized by a bearer token") — this is the next step of that thought. Vary puts the header in the cache key so a different token is a different entry.

The NS route doesn't need it (no-cache revalidates, authz re-runs). Cost is low in practice since the peer's real cache is IndexedDB with its own CID verification — the HTTP tier is a bonus, not the mechanism.

[bplatz]

This file is −377/+152 — a restructure plus two deliberate behavior changes (source_branch/branches from the wire, unparseable ledger_id degrading to verbatim rather than erroring the stream) — in live replication code with no test coverage. The six new tests cover the lifted pump, not this task.

You flag the gap and note it predates the PR; agreed on both. What I'd still suggest: either an integration test pinning the two deltas, or split the behavior changes into their own commit separate from the lift. As it stands a regression here has no failing test and no clean bisect boundary between "the refactor broke it" and "the intended change broke it."

[bplatz]

Suggest factoring the plan+execute body into one async fn that both arms call, so they differ only in the retry wrapper.

As written this is two copies of the main query entry path, and per the dev-dep feature unification (raised on #1714), cargo test -p fluree-db-api and every api bench compile the residency arm — the cfg(not(…)) production arm isn't compiled in those binaries at all. So nothing in the api suite exercises what ships, and drift between the arms would be invisible.

In #1714 the duplicated code was error-path only, which made this tolerable. Here it's the main path, so the exposure is larger.

[bplatz]

Wording: "exactly the trust posture * had" is not quite right, and the difference is the point of the change. * deliberately excludes Authorization, so cross-origin authenticated requests were impossible before and are possible now — strictly more permissive, deliberately.

Worth stating that way, because the consequence is real for internal deployments: a page the user visits can now reach an internal Fluree server with an Authorization header and read the response. Not CSRF — no ambient credentials — but it widens the browser-as-pivot surface.

Follow-up rather than this PR: cors_enabled is a bool with allow_origin(Any) hardcoded. Now that browser clients are a first-class use case, an origin allow-list is worth having.

[aaj3f]

Thanks — all four addressed, including the two you wanted settled before merge.

Vary: Authorization. Real bug, fixed. Your reasoning is exactly right and I'd only sharpen one part of it: private is doing less work here than it looks, because the browser's own cache is per profile rather than per principal, and immutable means it never revalidates — so the server never got a chance to refuse principal B at all. Applied to all three object responses (200, 206, and the 304, which refreshes the entry and so must carry the same key). Pinned in test_object_endpoint_immutable_caching_and_conditional_get and mutation-checked: reverting the source with the assertions kept fails the test. Left the NS routes alone for the reason you gave — no-cache revalidates and authorization re-runs server-side.

peer/subscription.rs coverage. Agreed on the substance. I took the test option rather than the commit split: post-rebase the split would be disruptive, and a test is the durable guard where a bisect boundary is a one-time convenience. Two tests at the conversion site that actually owns the deltas (server_sse.rs) — one pins source_branch/branches coming from the wire and that their absence still defaults cleanly so an older server stays readable; the other pins the unparseable-ledger_id degradation, with a precondition assertion that the fixture really is an id the canonical parser rejects, so the test cannot quietly stop exercising the fallback. The first is mutation-checked against the old hardcoded None/0.

The duplicated query entry path. Factored into one plan_and_execute_round; both arms now differ only in the retry wrapper. Your framing is what made this worth doing rather than merely tidy — given the dev-dep unification from #1714, the api suite compiles only the residency arm, so two copies meant nothing in that suite exercised what ships and drift would have been invisible. It is once per query rather than per row, and an async fn is a state machine rather than a boxed allocation, so there is no hot-path cost. Ordering inside the retry loop is unchanged (fresh tracker per round, floor charged, then plan+execute, guard and budget still outside the loop). All three configurations build; api suite 3319 green, residency retry 5/5.

CORS wording. Corrected, and in the code comment as well as the description — the claim was in both. It now says this is strictly more permissive than *, deliberately, because * excludes Authorization and so cross-origin authenticated requests were impossible before and are possible now, with the internal-deployment consequence spelled out and the not-CSRF distinction kept. The origin allow-list is noted as a follow-up rather than smuggled in here.

On the scope of your review: understood, and noted — the two new crates not being covered is worth someone's eyes before this lands, and I'd rather that be explicit than assumed.