Skip to content

feat: graph sync — whole-graph replacement committing only the delta - #1682

Merged
bplatz merged 8 commits into
mainfrom
feature/graph-sync-delta
Aug 26, 2026
Merged

feat: graph sync — whole-graph replacement committing only the delta#1682
bplatz merged 8 commits into
mainfrom
feature/graph-sync-delta

Conversation

@bplatz

@bplatz bplatz commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Problem

Maintaining a graph from an external authority — an ontology in an editor, a reference table regenerated by a pipeline — needs a full-replacement verb: submit the graph's desired contents and get one commit holding exactly what changed. Nothing does that today:

  • upsert derives its retraction set from the payload's (subject, predicate) pairs, so it structurally cannot retract what the payload omits.
  • CLEAR + INSERT churns every fact and hides the real delta in history.

Design

Sync is an Insert-shaped transaction carrying a graph directive (Txn::sync_graph). The payload is parsed by the normal JSON-LD insert parser and re-homed onto the caller-named target graph (a payload addressing named graphs itself is rejected — the scope is exactly one graph, never inferred from data). Staging adds a second wave after the upsert wave: every currently-asserted flake of the target graph is pushed as a retraction, and the existing mixed FlakeAccumulator nets retract+assert of the same fact to nothing. The staged set is therefore exactly current − payload retractions plus payload − current assertions; an identical payload stages zero flakes and takes the no-op path (no commit).

Because it rides the normal staging path, modify-policy, SHACL, uniqueness, the reifies-cascade, and novelty backpressure apply unchanged. The current-contents scan follows CLEAR's policy model (not view-filtered — an authoritative replacement must not leave rows the caller cannot see); whole-graph staging keeps CLEAR's memory profile, with chunked staging as the same known follow-up.

Blank nodes skolemize under a deterministic graph-scoped key, so exporters with stable labels resync bnode-rooted structures (OWL restrictions, lists) with zero churn; label-regenerating exporters still churn, and RDFC-style canonicalization is the designed follow-up.

Surface

  • Rust: Fluree::sync_named_graph(ledger, graph, data, {dry_run, allow_empty}){asserted, retracted, committed, t}; sync_named_graph_with(…, TxnOpts, Option<PolicyContext>); .sync_graph(iri, json) on both transact builders (execute / stage / build_commit terminals).
  • HTTP: POST /sync?ledger&graph[&dryRun][&allowEmpty] and /sync/{ledger}; JSON-LD; forwarded like every write in peer/Raft modes; dry runs stage under the real run's policy and inline-constraint inputs.
  • CLI: fluree sync <ledger> --graph <iri> [-f|-e|stdin] [--dry-run] [--allow-empty] [--json] [--remote]. Turtle converts to JSON-LD client-side. The source of desired contents is a seam (SyncSource) so R2RML-mapped Iceberg/CSV/spreadsheet sources plug in later as one variant, not a new command or endpoint.
  • Consensus: TransactionBody::JsonLdGraphSync wired into both appliers; RefTransactBuilder::build_commit now returns Ok(None) for a no-change write (the Raft worker republishes the current head, mirroring revert's NoOp) — this also fixes a pre-existing case where a zero-flake update/upsert under Raft would poison the queued request with EmptyTransaction.
  • CommitReceipt gains the assert/retract split so every verb can report both sides (Raft-applied receipts report 0/0; the applied-receipt wire format does not carry the split).

Safety rails: explicit scope (graph required), empty payload requires allowEmpty, dry-run pre-flight, target validation (absolute IRI, system graphs refused by shape) enforced at staging so every entry point meets it, BodyKind/TransactionBody variants appended last (postcard-positional in persisted Raft state; ordinals pinned by test).

Verification

  • it_sync_graph.rs (12): first population, identical-resync no-op, delta-only counts, dry-run, empty gate, graph scoping, stable-bnode resync, payload/system-graph/malformed-target guards, the policy-gated locked path, and the build_commit no-op — the sync wave, deterministic skolem key, and locked-path dispatch each proven load-bearing by mutation.
  • Server /sync contract test, CLI end-to-end flow through the binary, docs-coverage gate, body_kind_ordinals_are_append_only.
  • Full sweeps over core/transact/api/consensus/server all-features green (5,676/5,676); CLI suite 342/342; clippy -D warnings clean with test targets.
  • Docs: docs/transactions/sync.md, docs/cli/sync.md, endpoint/overview updates, and a Sync Contract section in docs/cli/server-integration.md for external servers supporting the CLI.

Deferred

Chunked staging for whole-graph ops; server-side Turtle payloads; RDFC bnode canonicalization; opt-in default-graph sync; mapped (R2RML) sync sources.

bplatz added 3 commits August 24, 2026 08:40
Maintaining a graph from an external authority (an ontology in an editor, a
regenerated reference table) needs a full-replacement verb: submit the
graph's desired contents, get one commit holding exactly what changed.
Upsert cannot do it — its retraction set derives from the payload's
(subject, predicate) pairs, so it structurally cannot retract what the
payload omits — and CLEAR + INSERT churns every fact.

Sync is an Insert-shaped transaction carrying a graph directive
(Txn::sync_graph). parse_sync_transaction re-homes the payload's templates
onto the target graph (a payload addressing named graphs itself is
rejected — the scope is exactly one caller-named graph). Staging adds a
second wave after the upsert wave: every currently-asserted flake of the
target graph is pushed as a retraction, and the mixed FlakeAccumulator
nets retract+assert of the same fact to nothing — the staged set is
exactly current − payload retractions plus payload − current assertions.
An identical payload stages zero flakes and takes the no-op path (no
commit), which now admits sync alongside update/upsert. The scan follows
CLEAR's policy model (not view-filtered — an authoritative replacement
must not leave rows the caller cannot see); modify-policy, SHACL,
uniqueness, reifies-cascade, and novelty backpressure all apply
unchanged. Whole-graph staging keeps CLEAR's memory profile; chunked
staging remains the known follow-up.

Blank nodes skolemize under a deterministic graph-scoped key
(sync + doc_scope(doc_id)), so exporters with stable labels resync
bnode-rooted structures (OWL restrictions, lists) with zero churn;
label-regenerating exporters still churn, and RDFC-style structural
canonicalization is the designed follow-up for them.

Surface:
- Fluree::sync_named_graph(ledger, graph, data, {dry_run, allow_empty})
  -> SyncGraphReport { asserted, retracted, committed, t }; an explicitly
  empty payload requires allow_empty (it clears the graph), and dry_run
  stages + counts without committing.
- Builder .sync_graph(iri, json) on both transact builders, riding the
  optimistic retry loop via a dedicated OpPlan.
- HTTP POST /sync?ledger&graph[&dryRun][&allowEmpty] (JSON-LD; bare and
  *ledger forms), forwarded like every write in peer/raft modes.
- Consensus TransactionBody::JsonLdGraphSync wired into both appliers
  (local and raft) with its own body-hash domain tag and BodyKind.

CommitReceipt now carries the assert/retract split (count_ops at commit
finalization) so sync — and every other verb — can report both sides;
raft-applied receipts report 0/0 (the applied-receipt wire format does
not carry the split).

Tests: it_sync_graph.rs pins first-population, identical-resync no-op,
delta-only commits, dry-run, the empty-payload gate, graph scoping,
stable-bnode resync, and the guards — with the sync wave and the
deterministic skolem key each proven load-bearing by mutation. Docs:
docs/transactions/sync.md plus endpoint/overview updates.

Follow-ups deferred: chunked staging for whole-graph ops, Turtle payload
support server-side, a CLI command, RDFC bnode canonicalization, opt-in
default-graph sync.
…der, no-op terminal, dry-run parity

Six defects in the graph-sync verb, all confined to its own plumbing:

- stage_under_lock had no SyncGraph arm, so its JSON-like fallthrough staged
  the payload as a plain TxnType::Insert into the DEFAULT graph and left the
  target untouched. That path is what Raft (build_commit_with_handle) and
  every policy-gated, pre-built, SPARQL and Cypher local commit take; the
  optimistic path the direct tests used hid it. Dispatch sync before the
  fallthrough; pinned by policy_gated_sync_targets_the_named_graph (fails
  with the original staging substituted back in).

- JsonLdGraphSync was inserted mid-enum in BodyKind and TransactionBody.
  BodyKind is postcard-encoded in persisted Raft state snapshots
  (QueueEntry.body_kind), where ordinals are positional, so every later
  variant shifted — existing snapshots and mixed-version nodes would decode
  Turtle/push/revert/etc. as the wrong operation. Both variants now sit last;
  body_kind_ordinals_are_append_only pins all thirteen ordinals.

- build_commit unconditionally built, so a no-change sync under Raft hit
  EmptyTransaction and poisoned the queued request. RefTransactBuilder::
  build_commit now returns Ok(None) for a zero-flake update/upsert/sync that
  registers no graph, and the commit worker republishes the current head
  with install: None (mirroring the revert NoOp short-circuit). This also
  covers the pre-existing zero-flake update/upsert case under Raft.

- The HTTP dry run staged with TxnOpts::default() and no policy, so it could
  leak whole-graph delta counts past modify-policy restrictions and report
  success where the real run would fail policy/SHACL/uniqueness. It now runs
  prepare_transaction_body, builds the governance-derived PolicyContext from
  the ledger state (the same builder the appliers use), derives TxnOpts via
  the new txn_opts_from_body helper shared with the consensus path, and
  stages through sync_named_graph_with.

- Only the api verb validated the target IRI; the committing HTTP path sent
  the query parameter straight through consensus, so relative/malformed
  identifiers could be registered and system-graph IRIs bypassed the guard
  on ledgers that never seeded them. Validation (absolute-IRI shape, the
  ledger's own #txn-meta/#config IRIs) now lives at staging, via
  fluree_db_core::graph_registry::validate_absolute_graph_iri, so every
  entry point meets it; pinned by staging_rejects_malformed_and_system_graph_targets.

- committed keyed on t > pre_t read separately from execution, so an
  unrelated concurrent commit could flip an identical sync to committed:
  true, and the dry run reported a t older than the state it examined. It
  now keys on flake_count first (a delta always carries flakes; the only
  zero-flake sync commit is registration-only, which advances t under a real
  id), and the dry run reports the staged snapshot's t.
…er contract

fluree sync <ledger> --graph <iri> [-f|-e|stdin] [--dry-run] [--allow-empty]
[--json] [--remote <name>] with the same policy flags as insert/upsert.

The target graph is the constant of the command; the source of the desired
contents is a seam (commands/graph_sync.rs::SyncSource). Today the source is
RDF text — Turtle is converted to JSON-LD client-side, so an ontology
editor's export works against any server implementing /sync (which is
JSON-LD only). Mapped sources (an R2RML mapping over an Iceberg table, CSV,
or spreadsheet) are designed to plug in as one more variant that resolves to
the same payload and flows through the same verb — no new command or
endpoint per source; --remote only moves where materialization happens.

Local runs call Fluree::sync_named_graph_with under the caller's policy
context; tracked/--remote runs POST /sync/{ledger}?graph=…[&dryRun][&allowEmpty]
via the remote client. --json emits the server's dry-run report shape on
both paths so scripts consume either identically; --dry-run --json is the
intended pre-flight for scheduled syncs (an unexpectedly large retracted
count is a cheap tripwire for a truncated export).

Docs: docs/cli/sync.md (+ README index and SUMMARY), and
docs/cli/server-integration.md gains the fluree sync --remote endpoint
bullet plus a Sync Contract section (parameters, auth bracket, required
semantics, dry-run response shape, error table) so external servers can
keep supporting the CLI.

Tests: a CLI end-to-end flow through the binary (first sync, no-op resync,
dry-run JSON, Turtle delta, allow-empty gate + clear) and a server HTTP
contract test for /sync (delta commit, no-op t, dry-run shape, the 400
guards, allowEmpty clear).
@bplatz
bplatz requested review from aaj3f and zonotope August 24, 2026 14:04

@aaj3f aaj3f left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really nice solution for a fairly common scenario we've faced for years, @bplatz, and I like several of the design/implementation choices you did to fold it into the existing code: a second retraction wave into the existing mixed accumulator is a really nice pattern, and it means the delta semantics come out of Flake's own identity rather than anything sync-specific.

I went at the netting from a few angles that the shipped tests don't cover (language tags, @list reordering, a real non-root modify policy, and every degenerate "looks empty" payload I could think of) and it held on all of them; the stage_under_lock dispatch fix and the append-last ordinal pin in the second commit are the two things that would actually have bitten under Raft, and both go red under mutation.

The one thing to maybe get in before merging is mechanical: Cargo.lock wasn't regenerated for fluree-db-cli's new fluree-graph-turtle dependency, so a clean checkout dirties itself on first build.

Beyond that my notes are a coverage gap (no sync test runs against index-resident data — the f.g stamp at stage.rs:818 is unpinned), a wire-contract question I'd like your call on (the committed /sync response can't tell a pipeline what changed or whether anything committed), and a few doc/nit items.

Since this is stacked on #1681 it has had no ci.yml run — I ran fmt, package-scope clippy -D warnings, and the transact/api/consensus suites locally in its place.

Adherence to repo commitments:

  • Patterns/abstractions: ✔ extends FlakeAccumulator + scan_graph_flakes + the upsert-wave precedent; models sync as TxnType::Insert + directive rather than a parallel pipeline; target validation lives at staging so every entry point meets it.
  • Performance (speed first, memory second): ✔ ordinary insert/upsert/update pay one per-transaction Option check; stage_graph_mgmt dispatch untouched; one new O(n) count_ops pass per commit (negligible, unbenched); sync's own O(|graph|) staging is the same cliff CLEAR GRAPH already exposes and the default graph is unreachable. No performance-degradation risk on the per-flake hot path.
  • Testing: ⚠️ 12 integration tests wired into grp_ledger and green, plus server contract and CLI e2e; skolem key and build_commit no-op are mutation-load-bearing; but nothing exercises index-resident data (the g stamp mutation stays green) and no non-root policy test (I probed one; it passes).
  • Conventions: ⚠️ multi-line, mechanism-first commit bodies; fmt + clippy clean; docs added (docs/transactions/sync.md, docs/cli/sync.md, server-integration contract); Cargo.lock not updated.

Verified locally at branch HEAD ad34872: cargo fmt --all -- --check clean; cargo clippy -p fluree-db-{transact,api,server,consensus,core,cli} --all-targets -- -D warnings 0 warnings; nextest -p fluree-db-transact 350/350; nextest -p fluree-db-api --test grp_ledger 157/157 with all 12 it_sync_graph::* by name; consensus body_kind_ordinals_are_append_only pass; 4 purpose-written probes green; 3 mutations (skolem key red, build_commit no-op red, g stamp green).

Approving so you can merge when ready, but do get the Cargo.lock line in first, and I'd like to settle the committed-response shape here before --remote scripts start depending on it.

Comment thread fluree-db-cli/Cargo.toml
fluree-db-novelty = { path = "../fluree-db-novelty" }
fluree-db-nameservice = { path = "../fluree-db-nameservice" }
fluree-db-core = { path = "../fluree-db-core" }
fluree-graph-turtle = { path = "../fluree-graph-turtle" }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking (small). Cargo.lock was not regenerated for the new fluree-graph-turtle dependency.

The PR adds fluree-graph-turtle = { path = "../fluree-graph-turtle" } to fluree-db-cli, but the committed Cargo.lock has no "fluree-graph-turtle" entry under fluree-db-cli's dependency list. The first cargo command on a clean checkout of this branch rewrites the lockfile (I saw M Cargo.lock after my first build, diff is the single line + "fluree-graph-turtle", at Cargo.lock:2613).

Consequence: ci.yml doesn't pass --locked so CI silently self-heals, but anything that does (cargo build --locked, cargo-dist release builds) fails, and every developer who touches the branch gets a dirty tree.

Fix: cargo update -p fluree-db-cli --offline (or just commit the one-line Cargo.lock delta a build produces).

Commenting here because Cargo.lock is not in this diff.

Comment thread fluree-db-transact/src/stage.rs Outdated
// `flake.g`, and the payload's assertions carry the graph
// Sid — both sides must agree for the unchanged-fact
// cancellation to fire.
f.g = Some(sync_graph_sid.clone());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional (coverage). This is more of an observation than a finding. The f.g = Some(sync_graph_sid.clone()) stamp has a comment saying both sides must agree for cancellation to fire — but when I removed it, all 12 it_sync_graph tests stayed green (recompile confirmed). Novelty-resident flakes already carry g from commit_flakes.rs:39, so the stamp's only job is flakes decoded from the binary index after a reindex, and no sync test reindexes. I couldn't establish in the time whether index-decoded named-graph flakes come back with g set; if they don't, this stamp is exactly what keeps an identical resync of an indexed graph from churning, and the suite can't tell the difference. One test that populates, reindexes, then resyncs identically and asserts committed: false would pin it. Minor and non-blocking — but if you agree it's right, I'd rather see it folded in now than lost in the backlog.

.into_response());
}

execute_transaction(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-server/src/routes/transact.rs:1328-1340optional (wire contract; needs your call). The committed /sync response is the standard transact response, so it carries no asserted / retracted / committed — only the dry run returns the report shape. The verb's whole point is "one commit holding exactly what changed", and a pipeline calling the real run can't learn what changed, or whether anything was committed at all, except by comparing t to a value it read earlier. The CLI mirrors this: graph_sync.rs:197-222 prints the raw transact JSON for remote real runs, so the commit message's "--json emits the server's dry-run report shape on both paths" holds locally only. I understand the reason (the Raft AppliedReceipt doesn't carry the split, so under Raft the counts would read 0/0), but committed is derivable on every path (flake_count > 0 || t advanced) and the counts are real on the local-committer path. I'd rather settle the shape now than after fluree sync --remote scripts depend on the raw one — this is the "needs a decision the author shouldn't make unilaterally" case, so happy to talk it through here rather than defer it.

graph_iri,
data,
txn_opts,
None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-api/src/admin.rs:1113-1121optional. I don't think this matters much, but the dry run stages with index_config: None, so the at_max_novelty backpressure pre-check (stage.rs:626-631) never runs on a dry run while the real run would reject with NoveltyAtMax. It slightly weakens the "dry run fails the way the real run would" promise. Passing the ledger's IndexConfig through would close it. Fold-in-now if you agree.

///
/// The transaction's insert templates must all target this graph (the
/// parser re-homes them) and the graph must be a user graph — reserved
/// system graphs and (for now) the default graph are rejected at

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-transact/src/ir.rs:218-219 (and fluree-db-api/src/admin.rs:1021) — nit (doc accuracy). Both docs say the default graph is "rejected at staging". There's no such check — it's unreachable because registry slot 0 has no IRI (graph_registry.rs:133) and nothing aliases it. That's fine today, but the wording implies a guard a future #default-style alias would silently bypass. Either an explicit DEFAULT_GRAPH_ID refusal in the sync_scan match at stage.rs:697-725 or rewording to "cannot be addressed (no IRI)" would make the doc true.

// temporal metadata current so the next commit's event-time guard and
// dual-stamp decision stay in-memory integer checks.
let head_temporal = HeadTemporal::from_commit(&commit_record).or(base.head_temporal);
let (assert_count, retract_count) = count_ops(&commit_record.flakes);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (perf, non-blocking). count_ops is a new O(n) pass over every commit's flakes, for every verb, to populate the receipt split. It's a single predictable branch per flake and is dwarfed by the CID hash and serialization of the same flakes, so I don't think it moves anything — flagging only because it is a new pass on the commit path and nothing benches it. If there's an existing loop in finalize_state_with_base it could piggyback on, that'd be free.

scanned = sync_retractions.len(),
"graph-sync retractions generated"
);
acc.push_retractions(sync_retractions);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-transact/src/stage.rs:791-828 — the design is the right altitude: a second retraction wave into the existing mixed accumulator, so netting is Flake's own (s, p, o, dt, m) identity nested per graph. I probed "Alice"@en vs "Alice" (1/1 both directions, tagged resync no-op) and @list [x,y,z] vs [x,z,y] (exactly 2 retract + 2 assert, identical list no-op) — both correct with no special-casing.

// view-policy filtered — sync is an authoritative whole-graph
// replacement, and a view-filtered scan would leave rows the caller
// cannot see in place, breaking "the graph now equals the payload".
// Modify-policy is still enforced on the resulting flakes below.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-transact/src/stage.rs:1074-1090 ordering — cancellation runs before enforce_modify_policies, so a non-root caller who re-supplies a protected row unchanged never triggers policy on it, while omitting it fails the whole sync atomically (probed with a #config-driven f:allow false on ex:ssn; dry run fails identically).

.get("@graph")
.and_then(Value::as_array)
.is_some_and(Vec::is_empty);
let mut txn = if explicitly_empty {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluree-db-transact/src/parse/jsonld.rs:192-199 — the allow-empty rail can only be reached by a literal "@graph": []; every degenerate payload I tried ({"@context":…}, id-only node, [], {}) is rejected by parse_insert with the graph untouched.

// Graph sync: dedicated staging (whole-graph retraction wave). Must
// be dispatched before the JSON-like fallthrough, which would
// otherwise stage the payload as a plain default-graph insert.
if let TransactOperation::SyncGraph { graph_iri, json } = op {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit 2 (97c1b2f) — the stage_under_lock dispatch fix and the append-last ordinal pin are exactly the two things that would have bitten in production and not in the optimistic-path tests; both are mutation-proven (build_commit → None goes red as EmptyTransaction when disabled).

Base automatically changed from fix/values-object-scan-constraint to main August 26, 2026 02:41
bplatz added 3 commits August 25, 2026 22:42
`fluree-db-cli` gained a `fluree-graph-turtle` path dependency for
client-side Turtle conversion, but `Cargo.lock` was never regenerated, so
the first cargo command on a clean checkout rewrote it. `ci.yml` does not
pass `--locked` and silently self-healed; anything that does — cargo-dist
release builds — would have failed at release time.
Graph sync only worked while its target was entirely novelty-resident.
Against an indexed graph it failed outright, and so did CLEAR, DROP, COPY
and MOVE — all five verbs share `scan_graph_flakes`, and none had a test
that indexed first. Two defects, both pre-existing on `main`:

The scan issued `RangeTest::Ge` with an empty match. The V3 range provider
implements only `Eq` and rejects everything else as unsupported, so every
whole-graph verb errored the moment its target was indexed. `Ge` only ever
worked on the genesis path, where non-`Eq` tests pass through unfiltered.
`Eq` with an empty match is the whole-graph scan on both paths — the
provider treats "nothing bound" as a full-index cursor, and the genesis
path matches an empty `Eq` against every flake.

Once the scan runs, index-decoded flakes come back with `g: None`
regardless of graph — only novelty-resident flakes carry it. Every caller
routes by `flake.g`, where `None` is the default graph, so retracting an
indexed named graph committed phantom retractions against the default
graph and left the target untouched: CLEAR reported a commit and changed
nothing, COPY merged into the destination instead of replacing it, and an
identical sync of an indexed graph retracted and re-asserted every fact.
The scan now attributes every flake to the graph it scanned, which retires
the sync wave's own stamp as a special case. The transfer path already
computed the source Sid and discarded it under an underscore.

Two tests, both against a real index build: an identical resync must stay
a no-op and a delta must commit only the delta; and CLEAR, COPY and MOVE
must empty, replace, and move exactly their targets. Restoring `Ge` fails
both with the provider error; dropping the stamp makes the resync retract
three facts and leaves CLEAR's target intact.
bplatz added 2 commits August 26, 2026 09:17
Staging a whole-graph operation — graph sync, CLEAR, DROP, COPY, MOVE —
materializes every currently-asserted flake of the target graph, so peak
memory scales with the graph, not the delta. The worst case is the most
innocent one: an identical resync of a large graph materializes everything,
nets to zero, and commits nothing, and no existing guard ever sees it —
`at_max_novelty` runs before the scan but measures only current novelty,
`NoveltyWouldExceed` measures only the surviving delta after
materialization, and fuel is charged after the scan returns its Vec. Past
roughly 10M flakes the failure mode is an OOM kill; under Raft it is the
branch-owning worker node that dies mid-stage.

`FLUREE_MAX_GRAPH_SCAN_FLAKES` (default 10,000,000; 0 disables) caps the
scan, mirroring the FLUREE_PATH_MAX_VISITED backstop pattern. The cap rides
`RangeOptions.flake_limit`, which the V3 provider's drain loop honors
mid-scan — so the backstop bounds what is materialized, not just what is
returned. Exceeding it fails with an error naming the knob. The limit is
read per operation, never per flake, so it can be changed at runtime.

The default cannot regress any working deployment: whole-graph verbs
errored outright on index-resident graphs until the previous commit, and
novelty-resident graphs are already bounded well below 10M flakes by
reindex_max_bytes.

Because the guard lives inside the scan and is env-driven, a dry run trips
it exactly the way the real run would — no dependency on the
`index_config` the dry-run path doesn't pass. The test pins the scan-vs-
payload asymmetry (populating past the cap succeeds; rescanning fails),
dry-run parity, CLEAR sharing the backstop, and 0 disabling it.

The streaming SPOT diff that removes the materialization entirely — memory
bounded by the delta, not the graph — is #1691.
`fact-01m0srrpntk3mxseprnpcvey08` (the reverted VALUES-to-FILTER-IN
lowering note) grew to 910 chars on its last update, and the repo-memory
lint caps blocks at 750. Compressed without losing either load-bearing
finding — the singleton-fold interaction that unanchors the block, and
the missing object_bounds that make it slower where it fires. Also
carries this session's new memories into the committed store.
@bplatz
bplatz merged commit a824bbf into main Aug 26, 2026
14 checks passed
@bplatz
bplatz deleted the feature/graph-sync-delta branch August 26, 2026 14:10
@bplatz bplatz mentioned this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants