feat(sql): pushdown lane M1/M2 — MIN/MAX, BIND, widening filters, partitioning, caps, block cache, class policies by bplatz · Pull Request #1785 · fluree/db

bplatz · 2026-09-03T19:34:26Z

Builds on #1777, now merged; the branch carries main. Everything here keeps the lane's contract: a statement is built only where the answer is exact or provably a superset with the exact check kept in the engine, terms are built in-engine, and anything else declines to the per-scan lane.

What lands

M1 leftovers

MIN/MAX of one variable in a grouped statement, each extreme read through its own materialization alias.
BIND inside the block: the statement returns the columns the expression reads and the engine computes the value per row before residual filters. Positional admission; a BIND in a UNION branch or OPTIONAL is not admitted.
Widening filters: STRSTARTS/STRENDS/CONTAINS of a constant and REGEX anchored on a literal prefix push a LIKE … ESCAPE '!'; the filter stays a residual, so LIMIT stays in the engine and a grouped query over it declines.
Vertical partitioning: an entity whose members come from several triples maps on one subject is one access per distinct table joined on the key columns. The per-scan lane had the same silent-empty gap (F10) and is fixed in rewrite.rs so the oracle is right.
Caps: an IN list above the key-set cap stays a residual on the lane, and a block VALUES above it declines the block to the engine (an oversized statement used to be a hard execute_plan error); outer chunking and the UNION cap are pinned.

M2

Naive timestamps: a timestamp column without a zone (every MySQL DATETIME, default Postgres timestamp) is read as UTC when its term is built, and that is now the stated contract for comparisons too: an xsd:dateTime literal pushes exactly against it, rendered naive, so LIMIT and grouped statements push on that shape. SQLite stores timestamps as text in whatever format was written, so it keeps whole-day bounds around the literal (>= '2024-01-10') with the exact check in the engine. (An earlier revision pushed a ±14h window instead; see the review thread.)
Block cache past one key set: once the outer side exceeds 2000 rows the branch is counted once and, when it holds at most 100k rows (and an estimated 64 MiB) and no more than four rows per outer row seen, fetched whole and hash-joined to every later batch in memory. FLUREE_SQL_PUSHDOWN_CACHE_ROWS and FLUREE_SQL_PUSHDOWN_CACHE_BYTES override the caps (0 rows keeps every batch seeded). A count or fetch the provider cannot run leaves the branch seeded.
Class policies over a column-derived rdf:type: the gate decides each (map, predicate) verdict per targeted class and the lane renders it on the class column (kind IS NULL OR NOT kind IN ('staff'), or kind IN ('guest')), where the whole block used to decline. A class constraint on such a map (?p a <kind/staff>) becomes kind = 'staff'.

Found on the way

The R2RML literal encoder kept xsd:dateTime/date/time as string-backed literals, so FILTER(?t = "…"^^xsd:dateTime) matched nothing on any graph source and either lane (ordering coerces, RDFterm-equal is a type error). Fixed in operator.rs; pinned in it_graph_source_r2rml.
The per-scan lane has no column-derived class support (constant class against rr:class only, ?s a ?t never emits a derived class). The class-constraint pin is lane-only for that reason.

Review round

A REGEX prefix ending in a backslash is no longer pushed as a literal LIKE prefix.
Subjects compare by template skeleton: order/{id} and order/{order_no} mint one IRI space and join on their key columns, with the column-class vet at the edge. The text comparison had resolved that block to Empty, zero rows where the per-scan lane returns the join. Templates with disjoint literal prefixes keep the Empty shortcut; anything else declines.
A widened LIKE is marked BINARY on MySQL; the fake endpoint's predicates are three-valued.

Numbers

sql_pushdown_lane_probe, 1M-row Postgres shop, median of 5, lane only:

shape	seeded per batch	with the block cache
50,000 VALUES against `customers` (100k rows)	380 ms	177 ms
5,000 VALUES against `customers`	38 ms	39 ms (counted, stays seeded)
50,000 VALUES against `orders` (1M rows)	238 ms	252 ms (counted, stays seeded)
5,000 VALUES against `orders`	25 ms	43 ms (one COUNT(*) on 1M rows)

The last row is the cost of the unbounded count; bounding it needs a derived-table node in the plan IR.

Verification

Every new shape is pinned three ways on the fake endpoint (exact statement, rows, routing stamp) and replayed against the per-scan lane; each set of pins was checked to fail with its change reverted.
Live replay on SQLite, Postgres 16 and MySQL 8 behind the bridge, including the naive-timestamp comparisons on all three, the class constraint (kind = BINARY 'staff' on MySQL), the shared-subject join and the temporal equality.
The Iceberg and R2RML suites that don't need containers pass; the testcontainer suites were not run. CI runs on this branch now that it targets main.

Follow-up: #1780 (per-scan lane column-derived classes)
Follow-up: #1784 (bounded block-size probe)

…ment Each extreme now reads its column through its own materialization alias (AccessInfo.output_names names the statement outputs feeding an access), so MIN and MAX of one column, or of a key, are distinct outputs instead of declining as "MIN/MAX column also a key or another extreme". Pinned ungrouped and per-group, and live on SQLite, Postgres and MySQL.

A BIND no longer throws the block back to the per-scan lane. The statement returns the columns the expression reads and the engine computes the value per row before the residual filters, so a filter over the bound variable still works. Admission is positional: the BIND reads only variables the block bound before it, nothing the statement joins or filters on reads the bound variable, and a BIND inside an OPTIONAL or a UNION branch is left to the engine. DISTINCT narrowing keeps the columns the expressions read; the grouped statement ignores binds its aggregates do not read. Pinned with a per-scan differential and live on SQLite, Postgres and MySQL.

…ment cannot run exactly A filter the plan cannot evaluate exactly no longer costs the whole filter: STRSTARTS, STRENDS and CONTAINS of a constant against a string column, and a REGEX anchored on a literal prefix with no flags, push a LIKE with its own escape character while the exact filter stays in the engine over the returned rows. A collation can only match more strings than a byte prefix, so the pushed predicate is a superset. Inside a conjunction the parts that cannot widen are dropped from the pushed predicate; a disjunction needs every part. The filter remains a residual, so LIMIT stays in the engine and the grouped statement declines. Live on SQLite and MySQL the case-insensitive LIKE returns the extra row and the engine narrows it back.

…ring triples maps An entity whose members come from several triples maps sharing its subject template used to answer no rows and send no statement on the pushdown lane, and no rows on the per-scan lane either (F10 in the virtual-dataset audit). The lane now resolves such an entity to one access per distinct table, joined on the subject's key columns, with maps over the same table and subject sharing one access; a class from one map and a member from another share the access too, and a foreign key into the entity joins whichever access shares the parent's row. Every member must have exactly one providing map when no map provides them all, or the block is left to the engine. Object terms now name the triples map their object map belongs to and the materializer keeps an encoder per map, so a second map's datatypes encode correctly. The per-scan lane's star formation now refuses to fuse a star no single map covers, emitting the members as separate scans joined on the subject, so the differential oracle answers these shapes correctly. Pinned with a partitioned mapping on the fake endpoint and live on SQLite, Postgres and MySQL.

…e key-set caps A VALUES block or an IN list inside the block above the provider's key-set cap is no longer rendered into one statement: the block still runs on the lane, the VALUES in the engine and the IN as a residual. Such a statement would have been rejected by the provider as a hard query error rather than a fallback, since the block's own key set is not chunked the way outer bindings are. One outer key row over the byte budget runs the block unseeded. Pinned: 2001 outer keys go out as two statements, and a UNION above eight branch combinations declines with its tracked reason.

…statement-size caps

The literal encoder kept xsd:dateTime, xsd:date and xsd:time values as string-backed typed literals. Ordering coerces a string against a temporal literal, so FILTER(?t > ...) worked, but RDFterm-equal treats the pair as a type error, so FILTER(?t = ...) and != matched nothing on any graph source and on either lane. Coerce temporal datatypes like the numeric ones; a fractional lexical now renders canonically (.5Z, not .500000Z).

…tamps An xsd:dateTime literal against a timestamp column with no zone was never pushed, so the whole table came back for the engine to filter. Whatever zone such a column was written in, its values lie within 14 hours of the instant they denote, so a window that wide around the literal keeps every row the exact comparison can. lower_superset now pushes it for =, <, <=, > and >= as a naive TIMESTAMP literal (the database converts nothing) and the exact filter stays a residual; != is not widened. SQLite stores timestamps as text, where a time-of-day bound orders wrong across the two separators the bridge decodes, so a new timestamp_is_text capability renders the bounds as whole days (>= '2024-01-09'), which a date literal against a naive timestamp column now renders on SQLite only. Pinned on the fake and live on SQLite, Postgres and MySQL.

…y set Every outer batch became a seeded statement, so 50,000 ledger subjects against a 100,000-row table meant 25 round trips carrying key sets, and a block nothing seeds re-ran once per batch. Once the outer side has passed one key set (2000 rows) the block source now counts the branch with one COUNT(*) and, when it holds at most 100,000 rows and no more than four rows per outer row seen so far, fetches it whole with no modifiers and hash-joins every later batch to it in memory. A larger block, or one still far larger than the outer side, stays seeded exactly as before; the row cap is FLUREE_SQL_PUSHDOWN_CACHE_ROWS (0 keeps every batch seeded). On the 1M-row Postgres probe, 50,000 VALUES against customers (100k rows) drops from 380 ms to 177 ms; 5,000 stays seeded at 39 ms (was 38); the orders table (1M rows) is counted and stays seeded, 43 ms against 25. The decision is measured rather than estimated: the count is one statement, and the outer size is what has already streamed in. Pinned on the fake for the three paths (cached, above the cap, caching off) and two new probe shapes for each table.

…edicate A view policy targeting a class, over a map whose rdf:type comes from a column, made the whole block decline ("policy not static"): for a deployment with policy always on, the lane was off for that map. The gate now decides such a (map, predicate) verdict once per class any view restriction is selected by, against the no-class baseline, and the lane renders the result on the class column: rows of a denied class drop out (kind IS NULL OR NOT kind IN (...), a row without a class keeping the default) or only rows of an allowed class stay (kind IN (...)). A subject-targeted policy, a map deriving classes from several columns or maps, and a policy on an OPTIONAL member still decline. The same reversal lets a class constraint (?p a <kind/staff>) on such a map become kind = 'staff' instead of declining, and a class outside the template empties the block without a round trip. The per-scan lane answers neither shape (it matches a constant class against rr:class alone, #1780), so that pin is lane-only; the policy pins compare with the per-scan lane, which derives row classes for policy. Pinned on the fake and live on SQLite, Postgres and MySQL.

…e decision, and the scan lane's derived-class gap

aaj3f

@bplatz the improvement/additions on top of #1777 make sense, and most of what I'd have (as a human) said about this are already described in the review on 1777. I'll provide Claude's detailed review below. If 1777 gets any significant rework and you want an additional review on this as a result, just let me know.

This delta is careful work, @bplatz, and most of what I went after held up under adversarial probing: the LIKE escaping (%, _, ! — the ! mutation goes red), the four class-policy polarities against the native evaluator (flipping IS NULL reddens the oracle), positional BIND admission, the F10 resolution rules with same_row sharing and Empty on differing templates, the 2001-key chunking, the UNION cap, MIN+MAX through separate aliases — and, the thing I checked hardest, the per-scan star split in rewrite.rs fires only where fusing already returned zero rows, so no previously-fused Iceberg star splits and it_graph_source_r2rml stays 66/66. The block cache never double-emits: each child batch is wholly seeded or wholly cached per branch, and the join key is the same function the seeded path uses.

The one thing I have to block on is small: the REGEX prefix extraction's metacharacter set omits the backslash, so REGEX(?n, "^A\\w") pushes LIKE 'A\w%' and silently drops every matching row on Postgres, SQLite and Trino — reproduced on the fake with the scan lane as oracle (lane [], scan ["n=Ada"]); MySQL is only saved because sql_string refuses backslashes. It's the one shape in the PR where the lane narrows the answer, and it's one character plus a pin.

Fold-ins alongside: the VALUES-above-cap story (the code declines the whole block; the test doc, commit body and sql.md:225 say it stays on the lane — one of them has to change); count_block/fetch_block errors aborting a query that #1777 answered seeded (map to Seeded, as the size-unreadable path already does); the cache cap being rows rather than bytes; and the vertical-partition join edges skipping the type check — the same class as #1777's join finding, which the fix there covers. Two are genuinely your call and I've named them as such inline: the unbounded COUNT(*) probe is your own #1784 (needs an inner-LIMIT node — a real scope decision), and the ±14h window buys nothing under the engine's naive-as-UTC semantics (temporal.rs:38-45; the docs page says so) while costing the most common timestamp shape its LIMIT and its grouped statement — either push exactly and make naive-as-UTC the contract, or document the hedge.

Also inherited: #1777's review asks for changes that rebase underneath this branch (the native-path planner and batch-copy regressions, the folded-OPTIONAL and join-type declines), and nothing ran in CI for this head — three levels deep in the stack — so every gate above is local. One pre-existing wrinkle to know about when running clippy locally: without --all-features, fluree-db-iceberg trips a dead-code error (env_switch_enabled, aws-only caller) that stops the changed crates from being linted at all; with -A dead_code the 23 pedantic denies are clean.

Adherence to repo commitments:

Patterns/abstractions: ✔ extends #1777's lowering, plan and materializer structures (output_names, lower_superset, resolve_parts, plan_for_cache, policy_pred); widening predicates correctly stay residual.
Performance (speed first, memory second): ⚠️ lane net positive (cache: 380 → 177 ms at 50k keys); unbounded COUNT (#1784) and the window's over-fetch are real costs; cache unbounded by bytes; Iceberg per-scan lane no regression; native untouched (the native regressions live in #1777).
Testing: ⚠️ 14 lane tests with fake goldens, routing stamps and the scan-lane oracle, mutation-verified; the REGEX backslash shape is unpinned; one case enshrines a known-wrong per-scan answer; the fake is two-valued on NOT; nothing ran in CI on this head.
Conventions: ✔ ten focused multi-line commits; docs updated; follow-ups #1780/#1784 filed by the author; fmt clean; clippy clean on the changed crates once the pre-existing iceberg dead-code item is allowed.

Verified locally at branch HEAD 235259613: it_sql_pushdown_lane 14/14, it_graph_source_r2rml 66/66 (+2 ignored), it_sql_graph_source 6/6 (--features sql,native,iceberg; live_bridge_* pass trivially with no bridge here); fluree-db-query/sql/tabular unit suites green; cargo fmt --all -- --check clean; cargo clippy -p fluree-db-api … --tests --examples -- -D warnings clean; throwaway ^A\\w case red at HEAD, removed; two mutations red then restored byte-for-byte; worktree clean.

Just be sure to get the backslash into the metacharacter set (with its pin) before this merges, and let's settle the two decisions inline — happy to talk through either.

aaj3f · 2026-09-03T20:56:23Z

+                    return None;
+                }
+                let (col, pattern) = self.string_column_and_literal(args)?;
+                let prefix = pattern.strip_prefix('^')?;


Blocking (correctness — the widened LIKE is not a superset). The metacharacter set here is r".^$*+?()[]{}|"; the backslash is missing. So REGEX(?n, "^A\\w") yields the literal prefix A\w and pushes LIKE 'A\w%' ESCAPE '!', which requires a literal backslash in the data. Same for ^\d, ^a\.b, ^ab\b, ^\pL.

Reproduced on the fake with the per-scan lane as oracle (case inserted into cases(), since removed): SELECT ?n WHERE { ?c ex:name ?n FILTER(REGEX(?n, "^A\\w")) } — statement carries "t0"."name" LIKE 'A\w%' ESCAPE '!', lane rows [], scan lane ["n=Ada"]. MySQL is saved by accident (sql_string declines any backslash, so the shape falls back); Postgres with standard_conforming_strings on, SQLite and Trino render the literal backslash and silently drop rows. This is the one shape in the PR where the lane narrows the answer.

Fix: add \\ to the set (or stop the literal prefix at the first metacharacter including backslash) and pin it with a ^A\\w-style case.

Fixed in ccaa032: the backslash ends the prefix like the other metacharacters, so ^A\\w stays a plain residual. Pinned against the per-scan lane; with the character removed from the set the lane returns no rows for n=Ada.

aaj3f · 2026-09-03T20:56:23Z

@@ -1249,6 +1510,13 @@ impl<'a> Lowerer<'a> {
        if out_rows.is_empty() {


Should-fix (the prose and the code disagree — fold in now). An oversized in-block VALUES declines the whole block: this decline("VALUES too large to push") is propagated at :444-446 and fails the block in lower_block, so it falls back to the per-scan lane. The test that pins it says exactly that ("an oversized VALUES in the block declines the lane"), but its own doc comment, commit b71bfc4a9 ("the block still runs on the lane, the VALUES in the engine") and docs/graph-sources/sql.md:225 all say the block stays on the lane with the VALUES in the engine.

Either implement it — drop the key set from the statement and join in the engine the way outer bindings are — or fix the three pieces of prose. The IN-above-cap residual claim is correct (:1568).

Prose fixed in db88f19: the block declines when its own VALUES exceeds the cap, the IN list above the cap stays a residual. The test doc and the docs page now say that.

aaj3f · 2026-09-03T20:56:23Z

                    let mut joins = Vec::with_capacity(self.resolved.branches.len());
                    for branch in 0..self.resolved.branches.len() {
+                        if outgrown && matches!(self.caches[branch], BlockCache::Untried) {
+                            self.caches[branch] = self.count_block(branch, &graph_ctx).await?;


Should-fix (fold in now — the fallback rule applied to the cache). count_block and fetch_block are optimization statements, but both sites here use ?, so a provider that cannot run that shape, or a transient error, now fails a query #1777 answered seeded. The "size unreadable" path already degrades to BlockCache::Seeded for a bad result shape; the error path should do the same with a debug log.

Fixed in a16ba10: a failed count or fetch statement, or a failed page, leaves the branch Seeded with a debug log. Materializer errors still propagate since they would recur on the seeded path.

aaj3f · 2026-09-03T20:56:23Z

+    /// seeded for the outer rows seen, is fetched once and joined in memory
+    /// in place of a seeded statement per outer batch (or, for a branch
+    /// nothing seeds, a re-run per batch).
+    async fn count_block(&self, branch: usize, ctx: &ExecutionContext<'_>) -> Result<BlockCache> {


Should-fix (performance) — already tracked as #1784, so naming the decision rather than re-raising it. Once the outer side passes one key set, every Untried branch is counted with a full SELECT COUNT(*) over the branch root with no cap, and the commit's own probe shows the cost: the 1M-row orders shape goes 25 ms → 43 ms (+72%) and then stays seeded, so the count bought nothing. The body says a bounded probe needs a derived-table node in the plan IR — that's a real scope call and #1784 is the right place for it. Two things worth knowing meanwhile: the docs' "an index-only scan on most tables" is optimistic, since the root carries the IS NOT NULL predicates from require(), so the count needs a covering index; and SELECT COUNT(*) FROM (SELECT 1 FROM … LIMIT cap+1) x is the shape that bounds it once an inner LIMIT node exists.

Deferred to #1784, on the m3 branch: the derived-table node it needs exists there for sub-SELECT, so the probe becomes a count over a LIMIT cap+1 subquery. Agreed on the covering-index caveat; the docs wording will change with that fix.

aaj3f · 2026-09-03T20:56:23Z

 }

+/// A branch's rows, fetched once and joined to every outer batch in memory.
+struct CachedBlock {


Should-fix (memory, fold in now). The cache cap is rows, not bytes: rows: Vec<Vec<(VarId, Binding)>> plus a second copy of every join key in index. 100k rows × wide TEXT columns is unbounded, where the seeded path never held more than one page plus one child batch. A byte budget beside cache_max_rows, checked in the fetch loop that already checks rows.len() > max_rows and bailing to Seeded, bounds it. (FLUREE_SQL_PUSHDOWN_CACHE_ROWS is read once per query in resolve_block — that part is fine.)

Fixed in a16ba10: a byte estimate from the materialized bindings (binding plus owned string heap) is checked beside the row cap in the fetch loop, 64 MiB by default, FLUREE_SQL_PUSHDOWN_CACHE_BYTES overrides. The join index still adds a copy of every key on top of that estimate.

aaj3f · 2026-09-03T20:56:23Z

+            declined: None,
+        },
+        Case {
+            name: "a predicate two maps provide, with no map providing the entity, declines",


Optional. This case pins the per-scan lane's duplicated answer (e=ada@example.org l=Ada twice, e=cy@example.org l=Cy twice) as the expected rows. A graph is a set of triples, so one solution per subject is the right answer; the duplication is a pre-existing per-scan bug, but pinning it here enshrines it as expected output. Maybe mark the oracle as known-wrong for this shape (and point at the per-scan fix) rather than asserting the duplicates — minor, but I'd rather it not become the thing a future fix has to "break".

Leaving the pin as is on this branch. The m3 branch establishes set semantics in both lanes (a triple two maps mint alike is held once), and re-pins this shape to one solution per subject there.

aaj3f · 2026-09-03T20:56:23Z

@@ -741,6 +749,15 @@ fn parse_pred(text: &str) -> Result<Pred, String> {
    if let Some(c) = text.strip_suffix(" IS NULL") {


Optional (the fake is two-valued on NOT). Pred::Not(p) => !p.eval(t, r) at :722 (not in this diff), and Pred::In returns false on NULL, so NOT (NULL IN (…)) is TRUE in the fake but NULL — row dropped — in real SQL. The shipped policy predicate sidesteps it with IS NULL OR, so nothing is wrong today, but the fake cannot catch a 3VL mistake: flipping the IsNull branch to IsNotNull let the NULL-kind row through the fake where every real dialect would drop it (the scan-lane oracle caught it, the fake didn't). A three-valued eval would make the goldens honest on their own. Commenting here because :722 isn't in the diff.

Done in b6110fa: Pred::eval is three-valued, NOT preserves NULL, AND/OR follow the truth tables, only TRUE keeps a row. Re-ran your flip and the fake now drops the NULL-kind row on its own.

aaj3f · 2026-09-03T20:56:23Z

@@ -348,7 +348,11 @@ pub(crate) fn render_literal(lit: &Literal, ty: FieldType, dialect: SqlDialect)
        Literal::Int(i) => is_numeric(ty).then(|| i.to_string()),


Optional, suspected rather than verified. MySQL's LIKE matches per collation element, and its docs note 'ch' LIKE 'c%' can be false under Czech-style contraction collations (ch is one element) — if so, LIKE 'c%' is not a superset of the byte prefix c there and the "a collation can only match more" argument fails. Not verified live (the words fixtures use the default collation). Cheap hardening: render the widened LIKE with BINARY on MySQL, as the lane already does for = and keys, which makes it exact there at no cost.

Done in 9c35b0f: the pattern is marked BINARY on MySQL, pinned in the renderer tests and the live MySQL cases.

aaj3f · 2026-09-03T20:56:23Z

            continue;
        }

+        if let Some(m) = mapping {


Praise, and the thing I checked hardest. This is the hunk that touches the Iceberg per-scan lane, so I traced whether any previously-fused star now splits: covered is false only when no single map has constant-predicate poms for every star member, rdf:type members never enter the coverage list (they're returned early as class_filter/type_var), and previously such uncovered stars materialized zero rows (F10) — so the split fires only where fusing was already wrong, covered stars still fuse, and it_graph_source_r2rml is 66/66. No Iceberg regression.

aaj3f · 2026-09-03T20:56:24Z

@@ -144,19 +168,23 @@ impl R2rmlPolicyGate {
    /// materialize each row's classes.


Praise. I went looking for a bypass in the class-policy reversal and didn't find one: static_verdicts decides per policy_classes (subclass-expanded upstream in build_policy_set) against a no-derived-class baseline that already includes the map's static rr:class, allows() merges tm_classes with row classes so a policy denying a map's constant class still yields Deny, and all four polarities (kind IS NULL OR NOT kind IN (…) / kind IN (…) / Empty / no predicate) match the native evaluator — flipping the IsNull branch turns the pinned oracle red.

…lane-m1 # Conflicts: # fluree-db-api/tests/it_sql_pushdown_lane.rs # fluree-db-query/src/r2rml/policy.rs # fluree-db-sql/src/plan_render.rs

The anchored-literal extraction stopped at `. ^ $ * + ? ( ) [ ] { } |` but not at `\`, so `REGEX(?n, "^A\\w")` read `A\w` as a literal prefix and pushed `LIKE 'A\w%'`, which needs a backslash in the data. Postgres, SQLite and Trino rendered it and dropped every matching row; MySQL only escaped because `sql_string` refuses any backslash. The one shape where the lane narrowed the answer. The backslash now ends the prefix like any other metacharacter, and the pattern stays a plain residual. Pinned against the per-scan lane; with the character removed from the set the lane returns no rows for `n=Ada`.

Two maps whose subject templates differ only in the placeholder's column name — `order/{id}` over orders and `order/{order_no}` over shipments — mint the same IRI whenever the values agree, and the per-scan lane joins them on it. The lowering compared the template text, so it took the two for different subjects and, since no row could carry every member, resolved the block to Empty: zero rows, no round trip, where the per-scan lane returns the join. The same text comparison declined the shape in `bind_var`'s template arm. Subjects now compare by skeleton, the template with its placeholders anonymized: one skeleton means one IRI space, and the placeholder columns join pairwise. Each pair is vetted with `same_class` where the edge is pushed, so a bigint key against a text key declines before `open` commits to the lane rather than failing in the renderer. `same_row` keeps the exact comparison it needs: one relation, one subject map column for column. Two templates whose literal prefixes cannot begin one another (`order/…` against `customer/…`) are still provably apart and keep the Empty shortcut; templates the lane cannot relate either way, and a ref-object-map parent in that position, now decline instead of answering empty. The fixture gains `shipments` (a bigint key sharing the order template) and a memo map minting order subjects from the text `order_ref`. Both shapes are pinned against the per-scan lane and replayed live on SQLite, Postgres and MySQL; with the text comparison restored the join shape declines.

… by bytes `count_block` and `fetch_block` are optimizations, but both propagated the provider's error, so a backend that cannot run the count, or a transient failure, failed a query the seeded path answers. A failed statement or page now leaves the branch `Seeded` with a debug log, as an unreadable count already did. Materializer errors still propagate: they would recur seeded. The cache was capped by rows only; a hundred thousand wide text rows is unbounded. A byte estimate from the materialized bindings — the binding plus the heap its string owns — is now checked alongside the row cap in the fetch loop, 64 MiB by default with `FLUREE_SQL_PUSHDOWN_CACHE_BYTES` to override. The join index adds a copy of every key on top of that estimate.

The lowering declines the whole block when the block's own VALUES exceeds the key-set cap, and the pinning test asserts exactly that; its doc comment and the docs page said the block stays on the lane with the VALUES in the engine. An IN list above the cap does stay a residual on the lane. The prose now says what the code does.

bplatz · 2026-09-04T02:25:52Z

Merged main (5f1ff9e) and folded in the review's fix-now items:

ccaa032 — the backslash now ends the REGEX prefix; ^A\\w stays a residual, pinned against the per-scan lane.
4b561c5 — the partition join-edge item led to a wrong answer: same_subject compared template text, so order/{id} against order/{order_no} resolved the block to Empty (zero rows) where the per-scan lane returns the join. Subjects now compare by skeleton, placeholder columns join pairwise with the same_class vet at the edge, disjoint literal prefixes keep the Empty shortcut, anything else declines. Fixture gains shipments and an order_ref-keyed memo map; both shapes pinned and replayed live on SQLite, Postgres and MySQL.
a16ba10 — count_block / fetch_block errors leave the branch seeded; the cache gets a byte budget (64 MiB, FLUREE_SQL_PUSHDOWN_CACHE_BYTES) beside the row cap.
db88f19 — the VALUES-above-cap prose now matches the code (the block declines).

Left as named: the unbounded COUNT probe stays on #1784; the naive-timestamp window is a product call I'll take separately.

…ctly The lane reads a `timestamp` column with no zone as UTC when it builds the term, and the docs say so. The ±14h window pushed for a comparison against such a column was therefore pure over-fetch: it kept the exact filter in the engine, which cost the most common timestamp shape (Postgres `timestamp`, MySQL `DATETIME`) its pushed LIMIT and its grouped statement, and hedged against a naive-means-local reading the engine does not have. Naive-as-UTC is now the contract on both sides of the comparison. A dateTime literal is exact against a zoneless column, and the renderer writes it naive (`TIMESTAMP '2024-01-10 00:00:00.000000'`) where the column carries no zone, so `=`, `!=` and the orderings push as they do against a zoned column, and a LIMIT above them pushes too. SQLite keeps such a column as text in whatever format its writer used, which no rendering compares exactly, so the renderer still refuses a timestamp literal there and the lowering keeps the whole-day bounds, now around the literal's own day rather than the widened span. Letting the literal render on SQLite turned the per-scan lane's own pushed equality into no rows for a value stored without fractional seconds, which is what pins the refusal. The five naive-column cases are re-pinned against the per-scan lane and replayed live on all three backends.

A pushed LIKE is a superset of the engine's byte-prefix test only if the pattern matches at least those strings. MySQL matches LIKE per collation element, and under a contraction collation (`ch` as one element) `'ch' LIKE 'c%'` is false, so the widening could drop a row the engine would keep. The pattern is now marked `BINARY`, as string equality and key sets already are, which makes the LIKE exact there at no cost.

`Pred::eval` was two-valued, so `NOT (NULL IN (…))` was TRUE and a NULL operand under `NOT` kept a row that every real dialect drops. The shipped policy predicate sidesteps that with `IS NULL OR`, but the fake could not catch a mistake in it: flipping that `IS NULL` to `IS NOT NULL` let the NULL-kind row through the fake, and only the per-scan oracle noticed. A NULL operand now yields NULL, `NOT` preserves it, `AND` and `OR` follow SQL's truth tables, and only TRUE keeps a row. With the same flip the fake now drops the NULL-kind row as the real backends do.

bplatz · 2026-09-04T02:49:53Z

Settled the two decisions and the optional items:

0d81029 — the naive-timestamp window is gone. Naive-as-UTC is the stated contract on both sides: a dateTime literal pushes exactly against a zoneless column, rendered naive, so LIMIT and grouped statements push on that shape too. SQLite keeps whole-day bounds around the literal's own day, since its text timestamps have no format to compare exactly. Re-pinned and replayed live on all three backends.
9c35b0f — a widened LIKE is marked BINARY on MySQL.
b6110fa — the fake's predicates are three-valued; the IS NULL → IS NOT NULL flip now drops the NULL-kind row in the fake as the real backends do.

The duplicate-answer pin stays as is here; the m3 branch's set semantics replaces it.

…lane-m3 Carries #1785 and the review round on it. Beyond the textual conflicts: the subject-skeleton rule is ported into `parts_of`, so a union-entity resolution over templates that provably never meet is skipped and one the lane cannot relate declines; `collect_col_eqs` descends into derived tables and UNION ALL branches so the column-class vet reaches every join the statement renders; the fake's new predicate arms (`AsciiOnly`, `ExprCmp`) are three-valued like the rest; the SUM-over-nullable top-k rule reads the aggregates `group_plan` now receives directly.

bplatz added 10 commits September 3, 2026 12:57

chore(memory): record the widening-filter rule, the F10 fix, and the …

ca4a45d

…statement-size caps

bplatz added enhancement New feature or request area:query Query execution, planning, fast paths, overlay, result formatting labels Sep 3, 2026

bplatz requested review from aaj3f and zonotope September 3, 2026 19:34

chore(memory): record the naive-timestamp window rule, the block cach…

2352596

…e decision, and the scan lane's derived-class gap

aaj3f approved these changes Sep 3, 2026

View reviewed changes

bplatz mentioned this pull request Sep 3, 2026

feat(sql): pushdown lane M3 — subject policies, expressions, sub-selects, unions, HAVING #1788

Open

Base automatically changed from feature/sql-pushdown-lane to main September 4, 2026 02:07

bplatz added 5 commits September 3, 2026 22:12

Merge remote-tracking branch 'origin/main' into feature/sql-pushdown-…

5f1ff9e

…lane-m1 # Conflicts: # fluree-db-api/tests/it_sql_pushdown_lane.rs # fluree-db-query/src/r2rml/policy.rs # fluree-db-sql/src/plan_render.rs

bplatz added 3 commits September 3, 2026 22:49

bplatz merged commit c6ba1e9 into main Sep 4, 2026
17 checks passed

bplatz deleted the feature/sql-pushdown-lane-m1 branch September 4, 2026 03:05

		@@ -1249,6 +1510,13 @@ impl<'a> Lowerer<'a> {
		if out_rows.is_empty() {

		@@ -741,6 +749,15 @@ fn parse_pred(text: &str) -> Result<Pred, String> {
		if let Some(c) = text.strip_suffix(" IS NULL") {

		@@ -348,7 +348,11 @@ pub(crate) fn render_literal(lit: &Literal, ty: FieldType, dialect: SqlDialect)
		Literal::Int(i) => is_numeric(ty).then(\|\| i.to_string()),

		@@ -144,19 +168,23 @@ impl R2rmlPolicyGate {
		/// materialize each row's classes.

Uh oh!

Conversation

bplatz commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What lands

Numbers

Verification

Uh oh!

aaj3f left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bplatz commented Sep 4, 2026

Uh oh!

bplatz commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bplatz commented Sep 3, 2026 •

edited

Loading