feat(sql): pushdown lane — one statement per GRAPH block over a SQL source - #1777
Conversation
…QL source
A GRAPH <sql-source> block whose shape the lowering can accept now compiles
to a single SQL statement instead of one scan per subject star with the
joins done in the engine.
- `fluree_db_tabular::plan`: a dialect-neutral relational plan (accesses,
typed predicates, inner/left joins, key sets, projection, order, limit)
and the provider capabilities that gate it; lives in the tabular crate so
the query and SQL crates share it without depending on each other.
- `fluree-db-sql::plan_render`: renders a plan for Trino, Postgres, MySQL
and SQLite against probed schemas; key-set column types are inferred from
the table column they are equated with; string equality is byte-exact
(`BINARY` on MySQL).
- `R2rmlTableProvider::{pushdown_capabilities, execute_plan}` with
declining defaults; the SQL provider implements them, the Iceberg
provider is untouched.
- `r2rml::sql_lane`: the lowering (entity graph, key-column joins from
rr:joinCondition or identical templates, IS NOT NULL on required columns,
exact-vs-residual filter classification, folded and left-joined
OPTIONAL, static VALUES key sets, policy-pruned mapping, seeds for outer
bindings) and `SqlBlockOperator`, which resolves at open and otherwise
streams the ordinary GraphOperator. Outer bindings are chunked into a
VALUES key set so the source does the semi-join. Terms are built in the
engine from the returned columns through the existing R2RML
materialization, so datatypes come from the mapping.
- Planner: an explicit `GRAPH <iri>` block is estimated without the default
graph's statistics — a predicate "known absent" from the default graph
ranked the block as empty and placed it first, so nothing could ever be
seeded into it on either lane.
- Policy gate: `static_verdicts` decides every (triples map, predicate) up
front for a lane that plans the whole block; subject-targeted and
column-derived-type policies decline to the per-scan lane.
- Tests: a fake Trino endpoint that executes single-table statements, table
and key-set joins, filters, ORDER BY and LIMIT over in-memory tables, so
golden SQL, rows, routing stamps and the per-scan lane as differential
oracle are all asserted end to end.
…ed tables in the pushdown lane Both lanes assume a subject template's columns identify one row, which R2RML does not require. Registration now probes every table with `SELECT 1 … GROUP BY <subject keys> HAVING COUNT(*) > 1 LIMIT 1` (also over the parent columns of foreign keys pointing at the map), reports repeats as `mapping_warnings` (HTTP response, CLI output) and stores the flagged tables on the source record. The pushdown lane refuses a statement over a flagged table with an error naming it; `allow_duplicate_subjects` (HTTP field, `--allow-duplicate-subjects`) accepts the duplicate rows instead. `fluree sql check <source>` re-probes live tables and updates the record. An unreachable endpoint skips the probe with a warning. Docs: the pushdown lane and the uniqueness rule in graph-sources/sql.md, the CLI flag and `check` subcommand, the endpoint fields.
- Tracked queries report every statement the lane sent under `sql` (Tracker::record_statement -> TrackingTally.sql -> TrackedQueryResponse.sql). - Probed column types flow into the lowering (R2rmlTableProvider::source_schema, LowerInput.schemas): a literal is pushed only when the SQL type carries its class, so xsd:dateTime compares exactly against timestamp with time zone and a numeric literal against a text column stays a residual filter. - A template key that cannot be a value of its column (order/abc over a bigint) makes the block empty instead of failing the query. - Goldens for ORDER BY DESC / OFFSET top-k and pinned decline reasons; the fake endpoint orders decimal strings numerically. - Live differential against SQLite through fluree-sql-bridge, gated on FLUREE_SQL_BRIDGE_URL and run by the sql-bridge CI job. - OperationReceipt::Transaction and AliasOutcomeKind::Success box their tally.
A grouped query over one SQL-source block (COUNT, COUNT DISTINCT, SUM, AVG, MIN, MAX; GROUP BY alone as SELECT DISTINCT) now runs as one statement. The hook sits before the fused aggregate, whose operator becomes the fallback so Iceberg sources keep their lane. SPARQL semantics are patched where SQL differs: AVG is pushed as SUM + COUNT and divided in the engine, an empty SUM reports 0, results take the mapping's datatype, and string keys or extremes need a byte-comparing dialect. HAVING/ORDER BY/LIMIT stay with the engine's operators; a top-k on an aggregate output is offered when no HAVING applies. RelPlan grows group_by, aggregate output expressions and ORDER BY on an output name; the fake endpoint evaluates them. Goldens pin each shape, the per-scan lane is the oracle, and the live SQLite differential replays them (SUM/AVG over a NUMERIC column decline there by design and are pinned).
… its own switch The planner offered ORDER BY ASC LIMIT k to the operator below only under FLUREE_R2RML_TOPK_ASC, a gate that exists for the per-scan R2RML lane (an ASC directive on a nullable column could hide an unread row). The SQL pushdown lane orders only on required columns, so it is safe for either direction. Offer both directions unconditionally and apply the switch in the scan's set_topk, which keeps the scan byte-identical.
A UNION inside a block runs one statement per branch combination (each branch joined with the rest of the block, capped at eight combinations), not a SQL UNION ALL: the branches may bind a variable from columns of different types and the term datatype must stay the mapping's. Each branch has its own key sets, join plan and residual filters, which now run inside the source over branch-homogeneous batches. A branch that can yield nothing sends nothing; an aggregate over a UNION stays with the engine's grouping. A SELECT DISTINCT directly over the block reaches the source through a new Operator::set_distinct channel (ProjectOperator forwards it, Sort and Filter swallow it). The statement is then SELECT DISTINCT over the columns of the projected variables plus what the join and residual filters read, where the dialect's string equality is byte equality; the engine's DistinctOperator stays authoritative. Goldens and the live SQLite differential cover both, plus ORDER BY ASC LIMIT as a pushed top-k.
…hat it found The live differential ran only over SQLite, the one backend whose strings and timestamps cannot misbehave. The sql-bridge job now runs a bridge over each of SQLite, Postgres 16 and MySQL 8 (both servers five hours off UTC), replays every lane case on each, checks the rows against the fake's pinned rows as well as the per-scan lane, and adds dialect cases over words/tags/ events tables: byte equality, code-point order, a string join, a string key set and constant subject, DISTINCT / GROUP BY / COUNT DISTINCT / MIN / MAX per dialect, a zoned filter inside the server's offset, and decimal lexical scale. live_bridge_backends_are_configured_in_ci fails if a backend is missing, so a skipped backend cannot read as a pass. What it found, each pinned by a case that fails with the fix reverted: - Postgres reads `TIMESTAMP '… UTC'` as a naive value in the session's zone. A zoned literal is now `TIMESTAMP WITH TIME ZONE '… UTC'` there and `TIMESTAMP '…+00:00'` on MySQL. - MySQL's default collation folds case, so every string literal and template key renders `BINARY`, one side of a string join does too, and the duplicate-key probe groups `BINARY`. Grouping cannot be forced binary (ONLY_FULL_GROUP_BY rejects `GROUP BY BINARY col`), so a new `string_distinct_is_binary` capability, false on MySQL, declines DISTINCT / GROUP BY / COUNT DISTINCT over string columns. - A bridge test pins the driver's MySQL session `time_zone = '+00:00'` default and the zoned round trip, which a UTC server never exercises. live_bridge_round_trip seeds its own table through the bridge and runs in the same job.
The planner offered sources the primary sort key alone for a compound ORDER BY … LIMIT, on the reasoning that the sort above re-selects the exact k. That holds for the R2RML scan, which only skips files that cannot hold the top-k and so streams a superset under ties. The SQL pushdown lane answers exactly k rows, so `ORDER BY DESC(?t) ?o LIMIT 10` was sent as `ORDER BY total DESC LIMIT 10` and returned the wrong ten among ties on `total`. A 1M-row Postgres run surfaced it. `Operator::set_topk` now carries the complete ordering. The R2RML scan keeps pruning on the primary key; the lane, block and aggregate alike, pushes a multi-key top-k when every key is orderable (a required column, a COUNT, a SUM) and otherwise leaves ORDER BY and LIMIT to the engine. The fake SQL server evaluates multi-key ORDER BY so the goldens can pin both outcomes; two of the new cases return wrong rows under the old behaviour.
…gres `sql_pushdown_lane_probe` times the lane suite's shapes on both lanes over a live Postgres source and fails on any row-set disagreement, so a run is also a differential at scale. Single-key top-k shapes among ties compare the sort key's values only.
bplatz
added
enhancement
…e and dialect facts
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
This is really good and no arguments with the designs on how to approach this, @bplatz — the seams are well-placed (a dialect-neutral RelPlan in the tabular crate, pushdown_capabilities/execute_plan with declining defaults so Iceberg is untouched, SqlBlockOperator deciding at open and otherwise streaming the ordinary GraphOperator), set_topk carrying the whole ordering is the right generalization of a contract the scan was silently relying on, and the differential oracle with routing stamps is the right way to enforce "decline beats approximate". The NULL semantics in particular hold up under a lot of adversarial probing, and the pins have teeth — dropping the IS NOT NULL emission reddens all 29 goldens.
The two things I'd strongly recommend a revisit on are the body's "nothing here changes behaviour for Iceberg sources or for ledgers", which is false twice on the native path, and four admitted shapes where the lane's answer differs from the per-scan lane's. Native first: the planner now estimates every GRAPH <iri> block with no statistics — a ledger's own named graphs included, whose predicates are in the cross-graph aggregate the planner reads — and a scratch planner test shows the join order flipping (estimate 1,000,000 vs 50; block placed second instead of first), which in single-db mode turns 50 seeded probes into 200,000 correlated executions of the named graph. And because admits is shape-only, every admissible GRAPH <iri> block on a native ledger now routes through SqlBlockOperator, falls back at open, and then deep-copies every batch whenever it binds ≥2 new variables, because GraphOperator orders its schema from a HashSet while the lane orders it by pattern — the fallback has to be free for the wrap to be acceptable there. Both are few-line fixes (keep the stats and neutralise only the absent→0 short-circuit; make GraphOperator's schema order deterministic and turn the permute into an error).
Then the lane-vs-oracle divergences, each reproduced on the fake endpoint with the per-scan lane as oracle and re-traced by me: a folded OPTIONAL with several members binds per column instead of as a unit (p bound, s unbound, where SPARQL unbinds both); a constant object inside a folded OPTIONAL lands in the required access's WHERE (1 row vs 4); a join over varchar↔bigint (or decimal↔varchar) is a hard InvalidQuery from the renderer after open committed to the lane, where the per-scan lane returns rows — the lowering never proves join types the way it proves filter types; and a pushed top-k on SUM over a nullable variable lets the database order NULL where SPARQL's empty sum is 0 (g=1 s=-8 vs g=2 s=0). The fixes are in the inline notes and are all local declines or one extra check.
Fold-ins alongside: the dataset-mode nameservice lookup the lane can never use, the duplicate-subject refusal that never fires for rr:sqlQuery-backed maps, the registration probe's unbounded full-table aggregate with no opt-out and fail-open on error, the uncapped tracked-statement retention, and the per-row allocations on the in-memory join path. Also: nothing ran in CI for this head (stacked two deep), so every gate above is local — it wants a retarget to main once #1759/#1769 land so the Linux lane and the new three-bridge differential actually run before merge. The lane is JSON-LD-reachable (parse/lower.rs:430 lowers graph clauses to GraphName::Iri) with exactly one JSON-LD case in the suite; one JSON-LD replay of the routing stamps would pin parity.
Adherence to repo commitments:
- Patterns/abstractions: ✔ extends
R2rmlTableProvider/Operatorrather than a parallel dispatch; the fused aggregate becomes the fallback;⚠️ the wrap is not gated on "this IRI is a graph source", and join-type proof is delegated to the renderer after the fallback decision. - Performance (speed first, memory second): ✖ CRITICAL — native plans flip for
GRAPH <own-graph>blocks (planner.rs:988) and every declined native/Iceberg block binding ≥2 vars pays a per-batch deep copy (sql_lane/mod.rs:348); lane row path otherwise acceptable with should-fix allocations. - Testing:
⚠️ 11 lane tests / 46 pinned shapes with a real differential oracle and mutation-verified pins; no planner test for the newGRAPH <iri>arm; four admitted shapes diverge from the oracle; nothing ran in CI on this head. - Conventions: ✔ ten thorough multi-line commits; docs across seven pages; fmt/clippy clean locally on every touched crate.
Verified locally at branch HEAD 073eaf327: cargo fmt --all -- --check clean; clippy -D warnings --all-targets clean on query/api/core/sql/tabular/cli (--features sql,native,iceberg), consensus (--all-features), server (--features raft); it_sql_pushdown_lane 11/11, it_sql_graph_source 6/6, it_graph_source_r2rml 65/65 (+2 ignored), fluree-db-query 1514, sql+tabular 41, consensus 340; 18-shape throwaway differential (six DIFFER, twelve SAME) and the scratch planner test, both deleted; two mutations restored; worktree clean.
Just be sure to get the two native-path fixes and the four declines in before this merges — happy to talk through any of them, and I'd rather see the fold-ins land here than in the backlog.
| // source entirely), and a predicate "known absent" there would rank | ||
| // the block as empty, placing it first and never letting bound | ||
| // outer values seed it. A variable graph name iterates this | ||
| // ledger's graphs and keeps its stats. |
There was a problem hiding this comment.
CRITICAL (performance, native — blocking). This arm passes stats = None for every GraphName::Iri, but StatsView.properties is the ledger's cross-graph aggregate (it is what the planner reads for default-graph triples too), so a ledger's own named graph had accurate estimates before and now every inner triple estimates at DEFAULT_PROPERTY_SCAN_SELECTIVITY (1,000,000).
Reproduced with a scratch planner test (deleted): stats :name = 200,000 and :email = 50, patterns { ?s :name ?n . GRAPH <g> { ?s :email ?e } } — HEAD estimates the block at 1,000,000 (pre-PR: 50) and places it at index 1 of 2 (pre-PR: 0). In single-db mode the fallback GraphOperator runs the named graph once per parent row (graph.rs:741), so that shape goes from 50 seeded probes to 200,000 correlated executions. Any native query that mixes default-graph triples with a GRAPH <own-graph> block is exposed, which contradicts the body's "nothing changes for ledgers". No planner test covers the new arm, so the suite stayed green.
The commit body's actual complaint is the known-absent → 0.0 short-circuit for graph sources. Keeping the stats and neutralising only that — compute estimate_branch_cardinality(patterns, stats) and fall back to the None estimate when it comes back 0.0 — fixes the seeding bug without touching native plans. Pin it with the two-pattern test above.
There was a problem hiding this comment.
Fixed in a6caa4c, and confirmed exactly as you described it — a scratch planner test reproduces 1000000.0 against 50.0 and the block moving to index 1.
One correction to the suggested shape: estimate_branch_cardinality ends in (running * non_triple_estimate).max(HIGHLY_SELECTIVE), so it never returns 0.0 — a known-absent predicate comes back as 1.0. A == 0.0 sentinel would never have fired. The demotion keys on the HIGHLY_SELECTIVE floor instead.
Both halves are pinned separately: iri_named_graph_block_keeps_ledger_statistics reddens to 1000000.0 if the stats are dropped again, and absent_predicate_in_an_iri_named_graph_reads_as_unknown_not_empty reddens to 1.0 if the demotion goes. The cost you and I are accepting is that a native named graph whose predicate is genuinely absent ledger-wide is no longer short-circuited into first place; that shape returns nothing either way, and it is a much rarer query than the one that regressed.
| if batch.schema() == self.schema.as_ref() { | ||
| return Ok(Some(batch)); | ||
| } | ||
| // The fallback `GraphOperator` orders its output from a set, so one |
There was a problem hiding this comment.
CRITICAL (performance, native + Iceberg — blocking). This permute branch is not a rare path — it is the common one for any declined block that binds two or more new variables, and admits (:58-63) is shape-only, so on a native ledger every admissible GRAPH <iri> block now goes through SqlBlockOperator, resolves to None at open, falls back to GraphOperator, and lands here.
Mechanism: SqlBlockOperator::new (:87-94) declares its schema in pattern order; GraphOperator::new derives new_vars by iterating a HashSet<VarId> (fluree-db-query/src/graph.rs:145-158, fresh RandomState per construction) and emits batches in that order (graph.rs:654). With ≥2 new vars the orders disagree with probability 1 − 1/n!, and then every batch is rebuilt: a Vec per column plus col.to_vec() cloning every Binding per row, for the whole stream. Reachable from native named graphs in single-db mode, every multi-variable Iceberg star, and every dataset named graph. SqlAggregateOperator::next_batch (aggregate.rs:553-565) has the same copy for the grouped fallback. Mechanism verified by reading; magnitude not benchmarked.
Fix: have GraphOperator::new compute new_vars in deterministic pattern order (the same Vec + contains dedup this operator uses), then make this branch an Internal error instead of a silent copy — the fallback must be free for the wrap to be acceptable on the native path. Commenting here because graph.rs:145-158 is not in this diff.
There was a problem hiding this comment.
Fixed in a6caa4c. GraphOperator::new collects new_vars in pattern order with a Vec + contains dedup, which is byte-for-byte what SqlBlockOperator::new declares, and the permute is now an Internal error.
The existing schema tests asserted with contains, so they could not see order at all — that is why this survived. The new ones assert the exact vector across 64 constructions (five new variables, so 1/5! per run by luck); restoring the HashSet reddens them with a visibly scrambled order.
One deviation: SqlAggregateOperator keeps its permute, with a comment saying why. Its fallback is a fused or generic aggregate tree whose schema follows the query projection, while the operator declares group_by then aggregates, so those orders genuinely differ rather than differing by accident — and each batch there is already grouped, so the copy is bounded by group count rather than by the row stream. Erroring there would have broken a legitimate path.
| if !filters.is_empty() { | ||
| return decline("filter inside a folded optional"); | ||
| } | ||
| for (pred, obj) in members { |
There was a problem hiding this comment.
Blocking (correctness — lane returns rows the per-scan lane and SPARQL do not). In the folded path every OPTIONAL member becomes an independently nullable column of the required access; there is no all-or-nothing. SPARQL's LeftJoin treats the optional group as a unit: if any triple of the group is absent for a row, every variable the group binds is unbound.
Reproduced on the shop fixture (order 12 has placed set and shipped NULL): SELECT ?o ?p ?s WHERE { ?o ex:total ?t OPTIONAL { ?o ex:placed ?p . ?o ex:shipped ?s } } — lane o=order/12 p=2024-03-01 s=, per-scan (and SPARQL) o=order/12 p= s=. The same assumption in aggregate.rs:293-296 ("a template's columns are null together") makes COUNT(?opt) / GROUP BY ?opt over a folded multi-column template over-count (traced, not executed).
Fix: in the folded path admit exactly one member whose object is a variable reading one column (a RefObjectMap already goes through the LEFT JOIN path at :985-1005) and decline otherwise; a self-LEFT-JOIN is the exact alternative and the renderer already supports it. docs/graph-sources/sql.md ("an OPTIONAL member of the same entity is a nullable column of the same access (no join)") documents the wrong behaviour and needs the restriction.
There was a problem hiding this comment.
Fixed in 8b7a52e, as a decline: only a single member with a variable object folds now. Your shop-fixture repro is a pinned case, and the mutation reproduces it precisely — order 12 comes back p=2024-03-01 s= against the per-scan lane\s p= s=`.
Restricting to one member also fixes the policy interaction you did not mention: Some(false) => continue left a hidden member unbound while its siblings still bound, which is the same all-or-nothing violation. With one member, hidden means the group`\s only variable is unbound, which is right.
docs/graph-sources/sql.md updated — it documented the folded behaviour without the restriction.
| if !self.exact_eq(&col, &class, &lclass) { | ||
| return decline("constant object type differs from the column"); | ||
| } | ||
| self.access_preds |
There was a problem hiding this comment.
Blocking (correctness). With nullable = true, bind_member still pushes a constant object's equality into access_preds[alias] of the required access (here, and the IRI arms at :837-846 and :912-919). The new-entity OPTIONAL path relocates its predicates into the ON clause (:1182); the folded path never does, so the OPTIONAL's constant filters the required rows.
Reproduced: SELECT ?o WHERE { ?o ex:total ?t OPTIONAL { ?o ex:placed "2024-01-05"^^xsd:date } } sends … WHERE … AND "t0"."placed" = DATE '2024-01-05' — lane 1 row, per-scan 4. With a variable alongside (OPTIONAL { ?o ex:placed "2024-01-05"^^xsd:date . ?o ex:shipped ?s }) the lane returns 1 row against 4.
Fix: decline constant objects (IRI or literal) inside a folded OPTIONAL. A CASE WHEN … THEN … END projection would be exact but isn't worth the renderer change.
There was a problem hiding this comment.
Fixed in 8b7a52e — constant objects (IRI or literal) decline inside a folded OPTIONAL. Your repro is pinned, and reverting the guard reproduces the 1-row-against-4 exactly.
| } | ||
| Ok(()) | ||
| } | ||
| (KeyShape::Column { col: l, class: cl }, KeyShape::Column { col: r, class: cr }) => { |
There was a problem hiding this comment.
Blocking (the correctness-preserving-fallback rule). bind_var unifies two KeyShape::Columns on RdfClass equality only, and the rr:joinCondition edges (:360-369, :971-1003) are emitted with no schema check at all. The renderer then refuses a mismatched pair — same_class at plan_render.rs:376-384 → SqlError::Unsupported → QueryError::InvalidQuery (graph_source/sql.rs:714) — but by then open has already committed to the lane, so the query hard-fails instead of declining.
Reproduced: (a) rr:joinCondition [ rr:child "order_ref" ; rr:parent "id" ] over varchar ↔ bigint (the common legacy-schema shape): SELECT ?i ?o WHERE { ?i ex:order ?o } → lane errors cannot join t0.order_ref (String) with t1.id (Int64), no statement sent; per-scan returns 2 rows. (b) a shared variable over decimal(10,2) and varchar columns: { ?o ex:total ?t . ?q ex:amount ?t } → lane errors, per-scan 2 rows. Note two text columns both of class Numeric would pass same_class and render string equality, so '99.5' ≠ '99.50' is a silent wrong answer rather than an error (traced only).
Fix: at every ColEq emission check same_class(field_type(l), field_type(r)) against the probed schemas and, for non-Str classes, that both columns carry the class natively — the literal_exact rule applied to both sides, exactly as lower_filter's var-var branch already does at :1334 — and decline otherwise. Cheap belt-and-braces: render the unseeded plan once inside resolve_block and turn any SqlError::Unsupported into a Decline, so no renderer refusal can surface after the fallback decision.
There was a problem hiding this comment.
Fixed in 8b7a52e — and this one had a second cause underneath it that is worth knowing about, because the check you proposed does not fire without it.
I added the same_class vetting over every ColEq the plan will render, and my new test still failed with the original hard error: cannot join t0.order_ref (String) with t1.id (Int64). The reason is candidate_sources, which collects only the triples maps whose predicates appear in the block. A rr:parentTriplesMap is reached through its foreign key, never through a predicate, so the parent table was never probed — field_type returned None for the parent key column and the check silently passed on the half it could not see. The renderer knew both types because it probes separately, which is exactly why the mismatch only ever surfaced there. candidate_sources now walks to parent maps transitively. Both halves are load-bearing: reverting either one restores the hard failure.
Worth flagging beyond this fix — the lowering has been half-blind to parent-column types generally, not only for joins, so anything else keyed on field_type over a parent column was reading unknown.
Your silent-wrong-answer variant needed a separate guard: two text columns both classed Numeric pass same_class (both are String), so bind_var now also requires both sides to carry the class natively, the literal_exact rule you pointed at, with Str exempt.
On the belt-and-braces suggestion, I put the check in the lowering rather than rendering the unseeded plan in resolve_block: render_plan wants alias-keyed schemas and a dialect that are not at hand there, and vetting the ColEq set covers every emission site regardless. same_class and collect_col_eqs moved to fluree-db-tabular::plan (both crates already depend on it) rather than being re-stated, so the lowering and the renderer cannot drift again — the drift is what caused this.
| if key.is_empty() { | ||
| continue; | ||
| } | ||
| if seen.insert(format!("{key:?}")) { |
There was a problem hiding this comment.
Should-fix (performance on the outer-row path; traced, not executed). A few per-row costs in the in-memory join that the per-scan lane doesn't pay: the key-set dedupe here builds format!("{key:?}") — a Debug string of a Vec<Literal> — into a HashSet<String> per outer row per batch; :523 and :612 do a linear position() scan of the child schema per row per seed / join var; on the returned-row side :698 clones the whole Vec<usize> of matching outer rows per returned row, emit_row (:653) allocates a Vec<Binding> per row, and join_key (:776-784) goes through iri_of_binding which allocates a String then an Arc<str> per row per join var; aggregate.rs:597-603 clones every column of every result page into by_name before split_page moves them.
None of these is blocking — the lane's row path is otherwise one lowering per query and one render per chunk, and materialization is identical to the per-scan lane — but hoisting the positions, hashing Literal directly (f64 by bits), and taking the aggregate columns by index are each small. The one structural cost worth a design note: an unseedable block (keysets_for → [None], :513-514) re-executes the full unconstrained statement once per child batch with no cross-batch cache, which happens whenever a shared variable is nullable, constant-mapped, fails literal_exact, or is unbound in a single outer row of the batch. If you agree these are right I'd rather see them here than in the backlog.
There was a problem hiding this comment.
Two of these folded in (4ac091c), the rest not yet.
Done: the key-set dedup no longer builds format!("{key:?}") per outer row — it keys on the literals through a wrapper supplying the Eq/Hash that Literal cannot derive over its f64 (doubles by bits, so two NaN keys are one key, which is right for a set that exists to avoid sending the same key twice). Seed and join-variable column positions are resolved once per batch instead of by scanning the child schema per variable per row.
Not done, and I would rather do these with a profile than by eye: the per-returned-row Vec<usize> clone, emit_row's per-row Vec<Binding>, join_key going through iri_of_binding, and the aggregate page clone into by_name.
The structural one — an unseedable block re-executing the full statement once per child batch with no cross-batch cache — I have left alone deliberately. It wants the design note you suggest rather than a fix in this PR.
| } | ||
| } | ||
|
|
||
| fn fallback(&mut self, child: BoxedOperator) -> BoxedOperator { |
There was a problem hiding this comment.
Optional (minor perf). where_plan.rs:2727 already clones the patterns into SqlBlockOperator::new; fallback() clones them again, so a block inside a correlated subplan rebuilt per parent batch (OPTIONAL/UNION) clones the pattern tree twice per rebuild on the decline path. std::mem::take(&mut self.inner_patterns) here is safe — the patterns aren't needed after the decision.
There was a problem hiding this comment.
Not taken for now — it is real, but it is a clone on the decline path and I would rather not touch the pattern lifetime while the decline path is the thing several other fixes in this round have just moved. Happy to fold it in if you feel strongly.
| if term_type == TermType::Iri { | ||
| return RdfClass::Iri; | ||
| } | ||
| if let Some(l) = language { |
There was a problem hiding this comment.
Optional. class_of keeps rr:language verbatim while seed_values compares RdfClass::LangStr case-sensitively (terms.rs:226), though exact_eq ignores case (:1389) — an uppercase rr:language "EN" in a mapping drops matching outer rows from the key set. fluree_db_core::normalize_lang_tag on the way in would line the three up. Also worth knowing (conservative, not wrong): bind_var requires identical template strings to join two template-keyed variables (:757), so cust/{customer_id} never joins cust/{id} and a FK expressed via template rather than rr:joinCondition always declines.
There was a problem hiding this comment.
Both noted, neither taken. The rr:language case mapping is a genuine inconsistency between class_of, seed_values and exact_eq; normalize_lang_tag on the way in is the right fix and I would rather do it with a test that pins all three than fold it in here. The identical-template restriction is conservative-not-wrong as you say, so I am leaving it.
Both belong in the backlog rather than this PR — say the word if you disagree and I will pull them forward.
| // `FLUREE_R2RML_TOPK_ASC` (default on): off is byte-identical to the | ||
| // pre-item-8 DESC-only scan. The planner offers ASC unconditionally so | ||
| // the SQL pushdown lane, whose ORDER BY columns are required by | ||
| // construction, is not tied to this scan's switch. |
There was a problem hiding this comment.
Praise. This is the part of the set_topk change I was most worried about and it's right: the scan keeps pruning on the primary key only (a superset under ties, so the Sort above stays authoritative), re-applies its own FLUREE_R2RML_TOPK_ASC gate so the nullable-ASC safety property survives the planner now offering ASC unconditionally, and the DESC path is byte-identical. set_distinct is equally tidy — only ProjectOperator forwards it, Sort/Filter/Bind/Dataset/Graph swallow it, and DistinctOperator is always still stacked above, so DISTINCT can be neither skipped nor doubled.
| same && self.literal_exact(col, class) | ||
| } | ||
|
|
||
| fn place_pred(&mut self, pred: Pred) { |
There was a problem hiding this comment.
Praise. The NULL semantics are the thing most likely to go wrong in a lane like this and they hold up under the differential: FILTER(?opt != …), NOT IN, and OR/AND/NOT over OPTIONAL-bound columns land in WHERE and agree with the per-scan lane; a filter inside a new-entity OPTIONAL goes to the ON clause (:1176-1181) including one that references a required variable, which is exactly LeftJoin(Ω1, Ω2, expr); IS NOT NULL guards every subject and object column of the LEFT JOIN side; ORDER BY ?opt LIMIT on a nullable column is not pushed; SELECT DISTINCT widened by a residual's columns is re-deduplicated by the engine. And dropping the IS NOT NULL emission in require reddens all 29 goldens plus two row oracles, so the pins have teeth.
…GRAPH schema by pattern
Two regressions on the native path, both from the lane's wrapper rather than
from the lane itself.
The planner estimated every `GraphName::Iri` block without statistics. But
`StatsView.properties` is the ledger's cross-graph aggregate — the same map the
default graph is estimated from — so a ledger's own named graph had accurate
estimates before and now rated every inner triple at the unknown-scan default.
With stats `:name` = 200,000 and `:email` = 50, the block
`{ ?s :name ?n . GRAPH <g> { ?s :email ?e } }` estimated at 1,000,000 instead of
50 and was placed second, and in single-db mode the fallback GraphOperator runs
the named graph once per parent row: 50 seeded probes became 200,000 correlated
executions.
Only the "empty" verdict fails to carry over, since the name may be another
ledger or a graph source whose contents these statistics do not describe. So the
estimate is taken with stats and demoted to the statless one only when it claims
the block is empty. Note the branch estimate clamps to HIGHLY_SELECTIVE, so the
sentinel is that floor, not zero.
Separately, `GraphOperator` derived its new variables by iterating a HashSet
seeded with a fresh RandomState, so with two or more of them the output schema
order differed run to run. `SqlBlockOperator` declares the same schema at plan
time and its parent resolves column positions against it, so the mismatch was
absorbed by rebuilding every batch — cloning every binding of every row for the
whole stream. That is the common path on a native ledger, where `admits` is
shape-only and every admissible `GRAPH <iri>` block routes through the lane and
declines at open. The operator now collects in pattern order, matching what the
lane declares, and the permute is an Internal error: the fallback has to be free
for the wrap to be acceptable there.
The aggregate lane keeps its permute, and says why: its fallback is a fused or
generic aggregate tree following the query's projection, so the orders genuinely
differ, and each batch there is already grouped.
The existing schema tests asserted with `contains` and could not see order, so
the new ones assert exact order across repeated constructions.
…red from the per-scan lane
Each of these is a case the lowering admitted and answered differently from the
lane it must agree with, so each becomes a decline. The fixed shapes are pinned
against the per-scan lane as the oracle, and every pin was watched fail with its
fix reverted.
A folded OPTIONAL bound its members per column. SPARQL's LeftJoin binds the
group as a unit — if any triple of the group is absent for a row, every variable
the group binds is unbound — but nullable columns of the required access go NULL
independently, so an order with `placed` set and `shipped` absent bound ?p and
left ?s unbound where both must be unbound. Only a single member is folded now;
several need a self LEFT JOIN the lowering cannot yet build. A single member is
also the case where a policy-hidden member behaves correctly, leaving the
group's only variable unbound.
A constant object inside a folded OPTIONAL pushed its equality into the WHERE of
the *required* access — only the new-entity path relocates predicates into ON —
so `OPTIONAL { ?o ex:placed "2024-01-05"^^xsd:date }` returned one order instead
of every order. Constant objects decline.
A join between two columns the database cannot compare — the legacy `varchar` FK
against a `bigint` key — was refused by the renderer, but only after `open` had
committed to the lane, so the query hard-failed where the per-scan lane answers
it. The lowering now vets every column equality the plan will render, against
`same_class`, the renderer's own predicate: it moves to the tabular crate beside
the plan it describes, so the two cannot drift.
That check needed `candidate_sources` fixed first. It collected only the triples
maps whose predicates appear in the block, so a `rr:parentTriplesMap` — reached
through its foreign key, not through a predicate — was never probed, its columns
read as unknown, and any check vetting a column type silently passed on the half
it could not see. It now walks to parent maps transitively.
Matching RDF classes are also not a matching comparison: the class is what the
mapping reads a column as, so two text columns both mapped xsd:decimal agreed
and rendered a string `=`, where '99.5' and '99.50' are different strings and
the same number. Both sides must carry the class natively, the rule a literal
comparison already follows.
A pushed top-k over SUM ordered on a NULL the engine had not yet mapped to
SPARQL's empty-sum identity of 0. Every dialect sorts NULL at one end, so a
group whose summed column is NULL throughout ranked at an extreme rather than
among the zeroes and the wrong k groups came back. A SUM over a nullable
variable keeps its ordering in the engine; the grouped statement still pushes.
The fixture gains a nullable numeric and a text-to-bigint foreign key, since no
existing shape could express these.
…ation, tracking and join path
The duplicate-subject refusal never fired for an `rr:sqlQuery`-backed map. The
probe records whatever name it saw, which for a query-backed map is the
synthetic alias, while the plan carries the query text — and the refusal matched
only `RelSource::Table`, so such a map was flagged at registration, warned
about, and then run on the lane anyway with the very row multiplicities the
check exists to refuse. Each flagged name now resolves to the source the plan
would use, and both variants are compared.
The uniqueness probe is a full-table hash aggregate whose only ceiling is the
request timeout, and it ran even when the source was registered with
`allow_duplicate_subjects`, where nothing consumes its verdict. It is skipped in
that case, which is also the opt-out for a table large enough that the probe
would fail the registration.
A probe that errored warned and left the table unflagged, so the lane ran
statements over a table whose uniqueness was unknown. Those tables are now kept
in `unverified_subject_tables` — separate from the flagged ones, because "not
known to be unique" is not "known to be duplicated" and the two need different
advice — and the lane refuses them with a message naming `fluree sql check`.
`record_statement` retained every statement with no cap. Outer bindings chunk at
2,000 rows and a statement may run to 1 MiB, so a tracked query over a large
seed set retained tens of MiB and echoed all of it in the response. The first 64
are reported and the rest counted, in a new `sql_elided`, so a truncated report
says that it is one.
On the in-memory join path: the key-set dedup keyed on `format!("{key:?}")`,
allocating and formatting a Debug string per outer row per batch, and now keys
on the literals themselves through a wrapper supplying the `Eq`/`Hash` that
`Literal` cannot derive over its `f64`. Seed and join-variable column positions
are resolved once per batch rather than by scanning the child schema per
variable per row.
Not applied: the report's `resolve_block` dataset-mode item. Its premise is that
a graph source is never a dataset member, so the lane can never proceed there
and the capability lookup is wasted. A `FROM <sql-source>` query does build a
dataset in which the source is a named graph and the lane serves it —
`it_sql_graph_source` fails outright with the suggested early return. The
membership test is also what keeps a non-member name away from that lookup, so
the existing order is already the cheap one. Comment corrected to say so.
…ackend guard on its own job The lane's differential replays every case against real SQLite, Postgres and MySQL through the bridge, so the two tables the last commit added to the fake — `notes`, a text foreign key against a bigint key, and `orders.discount`, a nullable numeric — have to exist in all three live fixtures too. Without them the three live runs failed on `relation "notes" does not exist`. Separately, `live_bridge_backends_are_configured_in_ci` keyed on `CI`, but the workspace `test` job also runs with `CI` set and is not the job that starts the bridges, so it asserted three URLs that were never meant to be present there and failed. It now keys on a marker the differential step sets beside those URLs. The guard still does its job: with the marker set and a URL missing it fails, and without the marker it skips. This failure is older than the lane's review fixes — it is on the merge commit that first gave this branch a CI run at all.
Both query surfaces lower a block to `Pattern::Graph { name: GraphName::Iri }`,
so every admission and decline in this file is reachable from JSON-LD, but every
case was written in SPARQL. That gap matters most for the folded-OPTIONAL rules,
which count the members of the optional group: JSON-LD can spell that group as
several node objects or as one node object carrying several predicates, neither
of which has a SPARQL counterpart, and a group that folds where it should
decline returns wrong rows rather than slow ones.
Six twins cover a firing star, a single-member optional that must still fold,
both multi-member spellings, a constant object, and an explicit `graph` clause.
Each asserts the routing stamp, agreement with the SPARQL spelling, and
agreement with the per-scan lane over the JSON-LD query itself.
That last check is the one that is an oracle. With the multi-member decline
reverted, both surfaces fold and return `order/12` with `placed` bound and
`shipped` unbound, so they agree with each other and disagree with SPARQL's
LeftJoin — surface-to-surface parity alone would have passed. Reverting either
folded-OPTIONAL decline now reddens the stamp and the oracle together.
Confirms what the twins were written to find out: JSON-LD's single node object
with two predicates does reach the lowering as two members, so the existing
member count covers it.
`cargo clippy --all --all-features` reaches four `TrackingTally` sites in fluree-db-consensus that a default-feature check never compiles, so adding the field left them behind. Like `policy_enforcement` and `sql`, the count is read-path only and a replicated transaction has none, so the mirror carries `None` and the round-trip test asserts it.
|
Thanks — this was a genuinely load-bearing review. Both criticals, all four correctness divergences and the fold-ins are in; details are in the threads. Summary and the two places I pushed back: Commits
Every new pin was watched fail with its fix reverted, and each mutation reproduced the exact numbers from your notes: One finding underneath your join-types note. Two things I did not do.
JSON-LD replay (your parity suggestion). Six twins over the shapes whose routing depends on parser grouping. Worth reporting the process: my first version compared JSON-LD rows to SPARQL rows only, and when I reverted the multi-member decline both surfaces folded and were wrong identically, so parity still passed — exactly the "two lanes agreeing is not evidence" trap. It now also compares against the per-scan lane over the JSON-LD query itself, and both mutations redden the stamp and the rows together. It answered the question it was written to ask: JSON-LD's single node object carrying two predicates does reach the lowering as two members, so the member count covers a spelling SPARQL cannot write. CI. Retargeted to The body's "nothing here changes behaviour for Iceberg sources or for ledgers" was wrong on both counts, as you said — I will correct it. Postgres and MySQL live fixtures are verified only by CI; I ran the SQLite one locally against a real bridge. |
2 participants
Summary
A SQL graph source today answers a
GRAPH <source>block one triples map at a time: eachR2rmlScanOperatorstreams its table through the bridge and the engine joins, filters, groups and sorts the rows itself. A database would do all of that in one statement. This PR adds a pushdown lane that lowers a whole block, and a grouped query over one, into a single relational plan at the provider seam, renders it per dialect, and streams the result back as terms built in the engine.Stacked on #1769 (which carries the three standalone fixes the work uncovered), itself on #1759. Nothing here changes behaviour for Iceberg sources or for ledgers.
What is pushed
rr:parentTriplesMapbecomesJOIN … ON), well-designedOPTIONAL(aLEFT JOINwhere the dialect supports it), exact typedFILTERs (numeric, date, zoned dateTime,IN, string equality where the dialect compares bytes),VALUES, constant subjects reversed through the IRI template, and outer bindings sent as aVALUESkey set so the database does the semi-join.LIMIT,ORDER BY … LIMITas a top-k on typed required columns in either direction,SELECT DISTINCTover the projected columns,UNIONas one statement per branch.GROUP BYwithCOUNT,COUNT(DISTINCT),SUM,AVG(sent asSUM+COUNT, divided in the engine at full precision),MIN/MAX, grouped top-k on an aggregate output. The result datatype is the mapping's, via the same accumulators the fused aggregate uses.Anything else declines the lane for that block and runs on the per-scan lane exactly as before: residual filters that cannot be expressed stay in the engine, an inexact push never widens or narrows the answer. Decline beats approximate is the rule throughout, and the differential below is how it is enforced.
Dialects
The bridge speaks the Trino protocol over Postgres, MySQL and SQLite, and
dialectchanges only rendering. The live differential against all three (see Verification) forced aPushdownCapabilitiesrecord per dialect:string_eq_is_binary,string_distinct_is_binary(false on MySQL, whose default collation folds case and whoseONLY_FULL_GROUP_BYrejectsGROUP BY BINARY col),string_order_is_codepoint(Trino and SQLite only; Postgres and MySQL order by collation, so stringORDER BY … LIMITandMIN/MAXof strings stay in the engine there),left_join, and key-set and statement-size caps. Zoned literals render asTIMESTAMP WITH TIME ZONE '… UTC'on Postgres (plainTIMESTAMPsilently drops the zone) andTIMESTAMP '…+00:00'on MySQL. Documented under "Where dialects differ" indocs/graph-sources/sql.md.Impact
Measured with
fluree-db-api/examples/sql_pushdown_lane_probe.rson Postgres 16 in Docker over ashopschema of 100k customers and 1M orders (10 orders per customer, indexes onorders(customer_id),orders(total),customers(country)), release bridge, median of 3. The probe runs every shape on both lanes and fails if the row sets differ.COUNT(*)ORDER BY DESC LIMIT 10GROUP BYcustomer,COUNT+SUM(100k groups)GROUP BYcountry,COUNT+SUM(20 groups)COUNTReading the table: where the per-scan lane already pushed something (a typed filter, a constant key) the two are equal, which is what "decline beats approximate" should look like. The joins are the lane's case: the per-scan lane rescans the inner table once per 1,000-row batch of the outer, which is the 94 seconds on a 20-group query. The 100k-group shape is bound by minting 100k IRIs, not by the statement.
Safety
sql_block_pushdownandsql_aggregate_pushdown; the suite pinsMustFire/MustNotFireper shape with the exact statement, so a test cannot pass by silently taking the per-scan lane.FLUREE_SQL_PUSHDOWN_LANE=0keeps every block on the per-scan lane;FLUREE_DISABLE_QUERY_FAST_PATHSdoes the same and is the differential oracle.mapping_warningsandfluree sql check, withallow_duplicate_subjectsas the override.execute_trackedreports the statements sent.Fixes found on the way
ORDER BY … LIMITwith a top-k source. The planner offered sources the primary sort key alone, which is sound for the R2RML scan (a superset under ties) but not for a source answering exactly k rows:ORDER BY DESC(?t) ?o LIMIT 10was sent asORDER BY total DESC LIMIT 10and returned the wrong ten among ties.Operator::set_topknow carries the whole ordering; the lane pushes a multi-key top-k only when every key is orderable and otherwise leaves theLIMITto the engine. The 1M-row probe surfaced this.IN, key sets and joins (BINARY), and Postgres dropping the zone on a plainTIMESTAMPliteral, both caught by the live differential.Verification
it_sql_pushdown_lane: 46 pinned shapes against a fake SQL server (exact statement, routing stamp, rows), each also run on the per-scan lane and compared; the aggregate, key-set, policy, duplicate-subject and tracking tests.sql-bridgejob, one bridge per database.fluree-db-queryunit suite,it_sql_graph_source,it_graph_source_r2rml,it_iceberg_policy, clippy and fmt.Not in this PR
Vertical partitioning (one subject across several maps on one table), superset filters kept exact in the engine,
BINDinside the block, the caps above their thresholds, subqueries,REGEX, policy pushdown for column-derived types, statistics-informed key-set and join-direction choices, and an Iceberg executor behind the same seam. The docs' comparison table lists what each lane pushes.