Serve gap-free sync state + Doc#pending? / #compacted_state_update - #36
Merged
Conversation
… (0.3.0) A pending struct (a gappy update whose causally-prior update is missing — e.g. legacy data recorded before the update_ready? gate existed) has an empty integrated state vector, but encode_state_as_update merges the pending bytes back in (yrs merge_pending_v1). Answering a peer's SyncStep1 with that state handed the peer content it couldn't integrate: it parked the same pending forever, and the empty-SV / non-empty-content mismatch drove endless resync traffic (a browser re-sending frames several times a second). Fix: handle_sync_message now answers SyncStep1 with integrated-only state. The enabling primitive is yrs's public WriteTxn::prune_pending, which takes the pending out; we rebuild the state into a throwaway doc, prune it, and re-encode, so the live doc is never mutated. A fast path skips the rebuild when nothing is pending, keeping the clean case zero-copy. Neutralizes existing poisoned server state on deploy — no migration — and the server's pending still heals if the missing dependency later arrives (only then is the content served). Also exposes the primitives: - Doc#pending? — holds un-integrable pending structs / delete set? - Doc#compacted_state_update — full state, gap-free, non-destructive. encode_state_as_update stays lossless for raw-update recovery. Tests: Rust unit tests for has_pending / integrated_update (strips pending, non-destructive, fast-path byte-identical, heals); Ruby pending_test.rb incl. the exact server-store poison repro proving SyncStep2 no longer poisons a peer; a real-Y.js Gap fixture added to the generator. Bumps yrby 0.2.3 -> 0.3.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turns the ad-hoc verifications into regressions and covers the branches the first pass missed: - Mixed doc (real integrated content + a pending struct): prune keeps the content and drops only the pending (Rust + Ruby). - Diff/sv path -- integrated_update(doc, peer_sv), the signature handle_sync_message actually calls -- excludes pending in a diff (Rust + Ruby). - Pending *delete set* (pending_ds): an orphan deletion parks and is stripped from served state (Rust + Ruby). - Channel-level integration: Y::ActionCable::Sync answers SyncStep1 from a poisoned store with gap-free state (sync_test.rb). - Thread safety: pending?/compacted_state_update hammered under contention. New real-Y.js fixtures: Gap::DEPENDENT_OTHER (a second-client gappy insert) and PendingDelete::UPDATE (a delete-set-only orphan). All other fixtures regenerate byte-identically. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deliverable 1 of the gap-free-sync plan. Fixes the endless-resync poison from legacy pending structs, and exposes the supporting primitives.
The bug
A pending struct (a gappy update whose causally-prior update is missing — e.g. data recorded before the
update_ready?gate existed) has an empty integrated state vector, butencode_state_as_updatemerges the pending bytes back in (yrsmerge_pending_v1). So answering a peer'sSyncStep1handed it content it couldn't integrate → the peer parked the same pending forever, and the empty-SV / non-empty-content mismatch drove endless resync traffic (a browser re-sending frames ~several times/sec).Verified with the exact repro (server holds a gappy update → fresh client syncs → client ends up
pending?, empty, looping).The fix
handle_sync_messagenow answersSyncStep1with integrated-only state. The enabling primitive is yrs's publicWriteTxn::prune_pending(takes the pending out and returns it) — so no upstream patch, no Snapshot/GC fragility. We rebuild state into a throwaway doc, prune it, and re-encode; the live doc is never mutated, and a fast path skips the rebuild entirely when nothing is pending (clean case stays zero-copy).Updateframes) is unchanged.New primitives
Doc#pending?— holds un-integrable pending structs / delete set?Doc#compacted_state_update— full state, gap-free, non-destructive. For when you persist/serve state yourself.encode_state_as_updatestays lossless for raw-update recovery.Tests
has_pending/integrated_update— strips pending, non-destructive (source keeps pending), fast-path byte-identical to direct encode when clean, heals after the dep arrives.pending_test.rb): detection + heal; compacted excludes pending whileencode_state_as_updatekeeps it; the exact server-store poison repro proving theSyncStep2reply no longer poisons a peer; SyncStep2 still delivers real content.Gapfixture added togenerate_fixtures.mjs(all other fixtures regenerate byte-identically).Bumps yrby 0.2.3 → 0.3.0.
🤖 Generated with Claude Code