Accept causal gaps - #56
Merged
Merged
Conversation
jpcamara
changed the title
jpcamara
changed the title
jpcamara
force-pushed
the
feat/accept-causal-gaps
branch
2 times, most recently
from
August 4, 2026 05:17
5f349b0 to
0b92155
Compare
Adds an opt-in mode where a causally-gapped update is recorded immediately
as a pending struct and acked, instead of being rejected for a resync. It
heals when its missing dependency arrives via that dependency's own reliable-
delivery retransmit. Default is unchanged (reject + resync).
The motivation is durability and latency: reject-and-resync returns custody
of a received edit to the sender until a resync completes, and pays an O(doc)
resync round trip for what is usually a transient reorder. Accept mode makes
the edit durable on arrival and heals with no round trip. Serving stays gap-
free in BOTH modes (handle_sync_message / compacted_state_update exclude
pending), so the safety invariant -- never hand a peer un-integrable content
-- is untouched. Accept mode changes only what is stored, not what is served.
The non-obvious part, surfaced while building this: it is NOT just deleting
the reject branch. update_advances? flips false->true only on the FIRST
pending struct, so a second gap on an already-pending doc reads as a
duplicate and would be silently dropped. The concern uses a lossless full-
state comparison (sync_gappy_adds_content?) for the gappy path instead, which
correctly distinguishes a new gap from a duplicate retry. A native
"adds any struct?" primitive could replace it later.
Two costs, both documented and required to run it safely:
1. The store must be lossless: on_load preserves pending, and compaction is
guarded with doc.pending? (compacted_state_update strips pending).
2. An open gap is silent, not a loud resync storm: each recorded gap is
logged at info, and pending depth should be monitored.
Off by default; existing behavior and all existing tests are unchanged. Ports
directly to the Y::Sync::Engine refactor (same branch logic).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lity, native primitive
Supersedes the boolean accept_causal_gaps flag with the complete design
discussed, so the whole architecture is reviewable in one place.
causal_gap_policy dial (default :reject, unchanged):
- :reject rebuild + gap check; resync a gap (current behavior)
- :accept_strict record a gap but withhold the ack until it integrates, so
the sender's retransmits keep an open gap self-signaling
- :accept ack-on-durable via an INVERTED write path: append + relay +
ack with no doc rebuild (O(1)-ish vs O(history)); dedup is
delegated to the store's idempotency
Serving stays gap-free under every policy (handle_sync_message /
compacted_state_update exclude pending) — the policy changes only what is
stored and acked, never what is served.
Store contract for accept modes (documented + reference store in README):
lossless on_load (preserves pending), idempotent append (content hash), and
compaction guarded with doc.pending? so an open gap is never compacted away.
Unhealable-gap repair loop: on join / SyncStep1 with an open gap, the server
solicits the missing dependency from that client (any live client that has it
heals the gap) — no separate strike subsystem. A truly-unhealable gap surfaces
via on_gap rather than healing.
Observability: on_gap class hook (fired with the doc key when a gap is
observed) plus an info log, replacing reject mode's resync-storm signal.
Hook errors are swallowed so observability can't break frame handling.
Native Doc#update_adds_content? (yrby core): true if an update adds any content
(integrated OR pending). update_advances? flips false->true only on the FIRST
pending struct, so a second gap on an already-pending doc reads as a duplicate;
the new primitive stays correct. The concern prefers it and falls back to a
full-state comparison on an older core. Rust unit tests included and passing.
All Ruby tests (38) and Rust tests (35, incl. 3 new) pass locally; rubocop,
cargo fmt, and clippy clean. Still a proposal, default :reject.
Engine port (Y::Sync::Engine, #55) not included: it's on an unmerged branch and
I won't stack PRs; the policy logic is factored to port mechanically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P1: a retry of a still-gappy update returned :applied and got acked, telling the sender to stop retransmitting before the update integrated — defeating :accept_strict's self-signaling. A gappy duplicate now returns :recorded_pending (unacked); only a ready duplicate is acked. P2: sync_observe_gap fired (log + on_gap) before sync_record_change, so a failing on_change logged and metriced a durable gap that was never persisted, re-inflating on every retry. Observe now runs only after the record succeeds. Both covered by new/strengthened tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017TeaovH2jyHARHJSyQ8Afo
jpcamara
force-pushed
the
feat/accept-causal-gaps
branch
from
August 5, 2026 18:56
0b92155 to
c0d5733
Compare
# Conflicts: # CHANGELOG-rails.md # lib/y/action_cable/sync.rb
A causally-incomplete update is now recorded and acked like any other (ack-on-durable). It parks as a pending struct, is never served, and heals when its missing dependency arrives, usually via the join handshake soliciting it from the next client. The write path no longer rebuilds the document per update: it appends, relays, and acks. Gone with the reject path: the causal_gap_policy option (never shipped), the accept_strict middle mode and its update_adds_content? native primitive (the write path no longer dedups, so the ext/ changes revert entirely), the per-update ready/advances gate, and the sync_log_gap_resync reject logging. A lost-ack retry records again; replay converges because CRDT apply is idempotent. on_gap and the info log surface an open gap at join/serve time. The store contract tightens: on_load must preserve pending, compaction must not run while doc.pending? (Y::Document already quarantines), and on_change must tolerate duplicate deltas.
jpcamara
changed the title
handle_sync_message answered a SyncStep1 with integrated-only state on the theory that a served pending struct poisons the peer. It does not: a peer parks it exactly as this doc did and heals it when the missing dependency arrives, which Y.js's own encodeStateAsUpdate relies on. And the healing mechanism was never the serve-side filter; it is the ack loop, since the missing dependency is an update its own sender still holds unacked and keeps retransmitting. The one place pending must still be excluded is compaction, where folding a log into a snapshot would freeze an un-integrable struct into the base state forever. compacted_state_update, integrated_update, the pending? readers, and Y::Document's quarantine all stay.
jpcamara
force-pushed
the
feat/accept-causal-gaps
branch
from
August 10, 2026 12:31
5c7812c to
93c02d8
Compare
# Conflicts: # CHANGELOG.md
The delivery-guarantees bullet argued against rejection, a test message asserted what does not happen, and the changelog described an option that never shipped. The changelogs keep recording the delta from the released behavior; everything else states the behavior as the behavior.
Four comments and a README line still described serving as gap-free or pending as never served, which stopped being true when serving went lossless: the test section header, the solicit test, sync_receive's doc, sync_observe_gap's doc, and the observability lead. Pending is served and travels; what stays true is that its content is invisible in the document until the dependency arrives, so they say that.
The changelog said compaction must not run while doc.pending?, which overstates it: Y::Document itself compacts while pending by folding clean rows and quarantining pending ones. The invariant is that an acked update must never leave durable storage before it integrates, so a fold must not drop a pending row into a gap-free snapshot.
Five comments and README passages still described update validation, gap rejection, and resync-on-gap: the concern's module header, the Scope section's acking and gap bullets, the ephemeral-documents walkthrough, and the acks section. All five now state the current behavior; the Scope gap bullet also gains the healing story and the compaction quarantine.
A batch holding a causal gap quarantined every row in it, including rows with no relationship to the gap: the clean-set retry filtered on the pending marker, and a fresh gap row is unmarked, so the retry failed and marked the whole batch. Independent rows then stayed pinned until the gap healed. The gapped path now folds everything integrable into state in one pass and judges each row against the folded result: a row the folded state cannot integrate cleanly carries the gap or builds on it and is quarantined; a row that is ready and adds nothing is fully captured and deleted. A healed gap folds out at the next compaction instead of waiting for its dependency to be joined by a fresh threshold of edits. A gap-only batch leaves state untouched.
Join and reconnect handshakes already have a client send everything beyond the server's integrated state, and the missing dependency's own sender retransmits it until acked, so the extra server-initiated SyncStep1 on a mid-session sync bought only a narrow window: a gap formed after connect, healed before anyone reconnects, by a client that happened to sync. A regular Yjs server sends no unsolicited SyncStep1s, and now neither does this one. sync_request_resync lost its last caller and is gone; observing an open gap at serve time stays.
Rewrite the causal-gap sections in plainer second-person prose: lead with what you get, frame the store contract as guidance for custom stores, and fold the on_gap bullet into the observability paragraph. Drop two leftover claims that updates are checked against durable state before recording; the write path records, relays, and acks.
jpcamara
marked this pull request as ready for review
August 12, 2026 01:10
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A causally-incomplete update, one whose causally-prior update the store hasn't seen, is now recorded, acked, and served like any other update. No setting, no mechanism for swapping; earlier drafts added a three-mode
causal_gap_policy, and none of it shipped anywhere.The reasoning, in two halves:
Receive: acking is the durability mechanism, rejection never was. The missing dependency behind any gap is an update its own sender still holds unacked, and at-least-once retransmission delivers it until the server records it. Rejecting the dependent update added no durability; it punished a different client for someone else's in-flight frame and forced a resync round trip. So the write path now appends, relays, and acks (ack-on-durable), with no per-update document rebuild at all.
Serve: lossless, like Y.js itself.
handle_sync_messagenow answers a SyncStep1 with full state, pending included, matchingY.encodeStateAsUpdate. A peer parks a served pending struct exactly as the server did and heals it the same way. The old integrated-only filter guarded against a poisoning that does not happen; the one place pending genuinely must be excluded is compaction, where folding the log would freeze an un-integrable struct into the snapshot, and that boundary stays:compacted_state_update,integrated_update,Doc#pending?,update_ready?/update_advances?, andY::Document's pending-row quarantine are all kept.Store contract (now documented as such):
on_loadmust preserve pending, compaction must not run whiledoc.pending?(the bundled store already quarantines), andon_changemust tolerate duplicate deltas, since a lost-ack retry records again (replay converges; dedup is the store's option, shown as a content-hash upsert in the README).Observability for an open gap: the join handshake solicits the missing dependency from other clients, an
infolog, and a newon_gaphook firing with the document key at join/serve time.Deleted: the reject gate, the
accept_strictmiddle mode, itsDoc#update_adds_content?native primitive, andsync_log_gap_resync. One Rust change remains: the SyncStep1 branch ofhandle_sync_messagedrops itsintegrated_updatecall for a direct lossless encode.Suite: 175 runs green (the serve tests now prove park-then-heal end to end at both the Doc and channel layers), rubocop and clippy clean.