SQL graph sources: R2RML over Trino-protocol endpoints, plus the fluree-sql-bridge sidecar - #1749
SQL graph sources: R2RML over Trino-protocol endpoints, plus the fluree-sql-bridge sidecar#1749bplatz wants to merge 15 commits into
Conversation
…raph sources A SQL graph source scans tables through any endpoint speaking the Trino client protocol (POST /v1/statement + nextUri pages): Trino, Starburst, PrestoDB, or a sidecar in front of another engine. Every page is one plain HTTP request, so nothing is stateful on our side and no database driver is linked into the binary. The engine's R2RML operator asks a provider for one table at a time — projection, conjunctive filters, optional top-k — so this crate renders exactly one SELECT per scan. Filters are pushed typed against a cached LIMIT 0 schema probe; a literal that cannot be rendered safely for the column's type is declined rather than guessed, since a mistyped comparison fails the whole statement in Trino and the in-engine FILTER stays the authority regardless. Top-k is not pushed: a null in a key or required column would consume LIMIT slots and break the superset contract. Type decoding covers Trino's JSON renderings (dates, precision timestamps, numeric-offset zones, exact decimals, base64 varbinary, NaN/Infinity). timestamp-with-time-zone columns are selected AT TIME ZONE 'UTC' so a named-region zone never reaches the decoder. The endpoint guard blocks only the link-local/metadata range and follows no redirects: loopback/private hosts are the sidecar deployment shape.
Adds GraphSourceType::Sql (f:SqlMapping) and routes it through FlureeR2rmlProvider: has_r2rml_mapping / compiled_mapping read the mapping reference per source family, and scan_table / table_row_count dispatch to a Trino-protocol client when the record is SQL-backed. The COUNT shortcut is exact for SQL (SELECT COUNT(*) WHERE key IS NOT NULL), where Iceberg can only answer it from manifest stats. Fluree::create_sql_graph_source mirrors the R2RML registration path: the mapping is compiled and stored in CAS, the endpoint is probed with SELECT 1 (a failure is logged, not fatal), and the record is published. Clients are cached process-wide keyed by the raw-config fingerprint, so a rotated secret behind an env var does not rebuild the client every query. A SQL source has no snapshot to pin. Its build watermark records endpoint, table and first-touch time, and the loadTable-cache precondition in verify_build_snapshot_integrity is skipped for it. The end-to-end test drives registration, a plain scan, a pushed typed equality, date decoding and the count shortcut against a fake endpoint and asserts the SQL actually sent.
rr:sqlQuery compiles to a deterministic alias (sqlQuery:<hash>) that stands in wherever a table name is expected, so the scan operator, caches and find_maps_for_table need no changes; a SQL source resolves the alias back to the query and scans it as a derived table. Iceberg-backed sources refuse such a mapping at registration instead of at first query. The server route mirrors /iceberg/map with the SQL config surface (dialect, protocol, catalog/schema, user, bearer or OAuth2, session properties) and guards the endpoint against the link-local/metadata range. The CLI gains fluree sql map|list|info|drop; list/info/drop share the mapped-source implementations with fluree iceberg, whose family predicate now includes SQL.
…ySQL and SQLite A standalone workspace (excluded from the root one, like testsuite-sparql) so sqlx and its three drivers never enter the fluree binary. It serves the statement/page subset of the Trino client protocol: POST /v1/statement describes the statement and starts streaming rows through a bounded channel; each GET nextUri serves one page; DELETE cancels; an idle reaper drops abandoned statements. Column types are reported in Trino's names, so the Fluree side is unchanged whether it talks to Trino or to this. Protocol tests run in-process over a real SQLite file. An env-gated test in fluree-db-api (FLUREE_SQL_BRIDGE_URL) drives the real client against a live bridge and skips loudly when unset. CI gains a job for the workspace.
'warehouse' read as if it meant something to the endpoint; it was only the graph source name, and it collided with the Iceberg command's --warehouse flag.
The dispatch check in scan_table / table_row_count re-ran the nameservice lookup on every call, which on a storage-backed nameservice is two object reads per scan — doubling a cost Iceberg scans already pay once. The decision (and the opened SQL source) is now memoized in the per-query session.
…ps SQL mappings Snapshot pinning, incremental materialization and tracking parse the record as IcebergGsConfig; a SQL record now gets a clear refusal naming the source instead of a config-parse error. Hard drop resolves the mapping blob through the family-aware lookup, so a dropped SQL source no longer leaks its CAS mapping. Also: a ledger-join test (FROM NAMED + GRAPH over the SQL source), and the live test takes FLUREE_SQL_BRIDGE_CATALOG/SCHEMA so it runs against a real Trino (verified on 483) and a Postgres-backed bridge (verified on 16).
The example used an object with a "graph" key, which the JSON-LD parser
reads as an ordinary node pattern and rejects ("Arrays in property values
not yet supported"). The working syntax — now pinned by the JSON-LD half of
ledger_and_sql_source_join_in_one_dataset — is the array form
["graph", <id>, <pattern>] with the source listed in "from-named".
Noted while verifying: a fromNamed dataset-local alias does not resolve to
a graph source ("Graph source '<alias>' not found"), and keying the alias
by the source id collides with the id's own registration; the full id via
"from-named" is the form that works. Both are pre-existing behaviors.
bplatz
added
enhancement
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz interested to learn more about the grounded-reality scenario that made you look into this, but that's purely an academic curiosity. Everything here makes sense and the work is good.
The one thing you may want to block on before merging: dialect::sql_string doesn't escape backslashes, which is a pathway for SQL injection into a MySQL-backed bridge (MySQL processes \ escapes in string literals under the default sql_mode, whereas Trino/Postgres/SQLite don't). A query-author-controlled FILTER value or subject-IRI key with a backslash breaks out of the string literal and bypasses the mapping's table boundary. It's a small, localized fix in the bridge, plus a MySQL-backed regression test.
Adherence checklist
- Patterns / abstractions ✔ — extends
FlureeR2rmlProvider/R2RML operator,LogicalTable::SqlQueryalias unifies query-vs-table, Iceberg providers refuse SQL aliases; no parallel construct. - Performance ✔ — engine flake loops untouched; per-scan (not per-row) rendering; streaming both sides; per-query client/dispatch/schema caches reduce overhead. One note:
row_countissues a real backendCOUNT(*). - Testing
⚠️ — strong hermetic coverage for the client (wiremock) and the SQLite bridge; no MySQL/Postgres bridge tests, which is exactly where the injection hides. - Conventions
⚠️ — clippy-clean on base lints, thiserror error idiom, self-describing commits; butfluree-db-sqlis missing[lints] workspace = true. And: stacked ⇒ no CI ran — retarget to main before merge.
| } | ||
| } | ||
|
|
||
| fn sql_string(s: &str) -> String { |
There was a problem hiding this comment.
Blocking — sql_string does not escape backslashes → SQL injection on MySQL-backed sources.
String literals are rendered here by wrapping in '…' and doubling embedded ' only; backslash passes through unescaped. That's safe for Trino, Postgres (default standard_conforming_strings=on), and SQLite — none process backslash escapes in string literals. MySQL, under its default sql_mode, treats \ as an escape character inside string literals. The bridge runs the engine-rendered SQL verbatim (fluree-sql-bridge/src/mysql.rs:155, sqlx::query(&sql).fetch(&mut *conn), no NO_BACKSLASH_ESCAPES — grep of the crate confirms none). So for a dialect=mysql source, a value like x\ renders as 'x\' — MySQL reads \' as an escaped quote, the string never closes, and the following SQL is parsed as code. The tainted value is query-author-controlled: render_literal pushes Literal::Str(s) (a FILTER on a String column) and Literal::TemplateKey(raw) (a subject-IRI value reversed through a subject template) straight through sql_string (:319, :361). And dialect=mysql isn't exotic — it's required for MySQL, because it selects backtick identifier quoting.
Scenario: a user with query access to a MySQL-backed SQL graph source issues … FILTER(?name = "a\' UNION SELECT secret, … FROM other_table -- "). The bridge hands MySQL SELECT … WHERE + "name" + = 'a\' UNION SELECT secret, … -- ', reading a table the R2RML mapping never exposed. (Stacked ; statements are likely blocked by sqlx's single-statement execution, but UNION/subquery/blind extraction is not.) The mapping-as-access-boundary is defeated, up to the bridge DB user's privileges.
Preferred fix (one place, canonical): make MySQL obey the standard-SQL string rule the renderer already assumes. In fluree-sql-bridge/src/mysql.rs, set NO_BACKSLASH_ESCAPES on every pooled connection:
let pool = MySqlPoolOptions::new()
.max_connections(max_connections)
.after_connect(|conn, _meta| Box::pin(async move {
conn.execute("SET SESSION sql_mode = CONCAT(@@sql_mode, ',NO_BACKSLASH_ESCAPES')").await?;
Ok(())
}))
.connect(url).await…Then sql_string's quote-doubling is correct for all four dialects. Alternative (engine-side defense-in-depth): make sql_string/render_literal dialect-aware and double \→\\ for SqlDialect::Mysql — but that couples the renderer to MySQL's default mode and over-escapes if NO_BACKSLASH_ESCAPES is ever set, so the bridge-side fix is the clean one. Either way, add a MySQL-backed test that pushes a string filter containing \ and ' and asserts the injected value is treated as data — the current bridge tests are SQLite-only, the one backend this can't hit.
(The fix lives in fluree-sql-bridge/src/mysql.rs; commenting here on dialect.rs:287 because this is the render site that produces the unescaped literal.)
| futures.workspace = true | ||
| async-stream = "0.3" | ||
|
|
||
| reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] } |
There was a problem hiding this comment.
Optional, fold-in — add [lints] workspace = true.
This new library crate omits the workspace-lints opt-in that siblings fluree-db-r2rml and fluree-db-tabular both carry, so it silently forgoes the 23 workspace-denied pedantic clippy lints (uninlined_format_args, semicolon_if_nothing_returned, …). It's clean on the base lints today, but nothing keeps it that way.
Append:
[lints]
workspace = trueand re-run cargo clippy -p fluree-db-sql --all-targets -- -D warnings.
| }; | ||
| } | ||
|
|
||
| if let (Some(url), Some(secret)) = (&req.oauth2_token_url, &req.oauth2_client_secret) { |
There was a problem hiding this comment.
Optional (decision-needed, so fine to defer with the reason named) — secrets from the HTTP route are persisted as literal ConfigValues.
build_sql_config wraps auth_bearer/oauth2_client_secret in SqlConfigValue::Literal, so a secret submitted to POST /v1/fluree/sql/map is stored inline in the graph-source record config. The config layer supports env_var/secret_ref indirection (config.rs:76), but the HTTP path can only store literals — the secret lives at rest in the nameservice record. This matches the Iceberg REST catalog's model, so it's not a regression.
Worth a one-line note in docs/graph-sources/sql.md that the record carries the credential and should be protected accordingly — or a follow-up to accept a secret_ref through the route (a product call on whether the route should take refs).
| } | ||
|
|
||
| /// Exact `COUNT(*)` with the given columns required non-null. | ||
| pub async fn count(&self, source: &LogicalSource, non_null_cols: &[String]) -> Result<u64> { |
There was a problem hiding this comment.
Awareness, not a change request — row_count runs a real backend COUNT(*). Cardinality is answered with SELECT COUNT(*) … WHERE c IS NOT NULL AND … against the endpoint, so on a large SQL table it's a full aggregate scan per cardinality request, where an Iceberg source uses manifest metadata. Fine as an estimate, and the backend may optimize COUNT(*) — just a real query cost the planner should know it's paying if cardinality is requested repeatedly.
…rces # Conflicts: # Cargo.lock # fluree-db-api/Cargo.toml # fluree-db-api/src/graph_source/cache.rs
`sql_string` rendered literals with standard-SQL quote doubling and passed backslash through. That is the whole escaping rule on Trino, Postgres and SQLite, but MySQL's default `sql_mode` reads a backslash as live inside a literal, so a value ending in one escaped its own closing quote and the remainder parsed as SQL. Both `Literal::Str` (a FILTER on a String column) and `Literal::TemplateKey` (a subject IRI reversed through its template) carry query-author text, so a `dialect: mysql` source could be made to read tables its R2RML mapping never exposed. Fixed at both ends, because they cover different deployments: - The bridge sets `NO_BACKSLASH_ESCAPES` on every MySQL session it opens, so the standard rule the renderer assumes is the rule the server applies. - The renderer declines the pushdown for a MySQL string carrying a backslash rather than escaping for a mode it cannot observe — `dialect` names the database *behind* an endpoint that need not be a bridge we configured. Declining is the same valve type mismatches already use: the in-engine FILTER is authoritative, so it costs I/O, never correctness. Also here: - MySQL- and Postgres-backed bridge tests, gated on service-container URLs with a guard test that fails if CI stops supplying them. SQLite is the one backend whose literals cannot misbehave, so it could not cover this. - `[lints] workspace = true` on fluree-db-sql, which had been forgoing the workspace-denied pedantic set (3 findings, fixed). The bridge is a separate workspace and cannot inherit, so it mirrors the table. - Docs: the escaping rule and its pushdown decline, that `COUNT` is a real backend aggregate scan, and that both registration paths store credentials as literals in the graph-source record.
Same class as the MySQL fix, one file over. `standard_conforming_strings` has defaulted to `on` since 9.1, which is why Postgres was not exposed the way MySQL was — but it remains a settable GUC, and a server, database or role with it `off` would process backslash escapes inside ordinary literals. Setting it per session costs nothing and drops the dependency on server configuration. Asserts both session settings directly. That assertion is the whole fix for MySQL (a default server does not set NO_BACKSLASH_ESCAPES, so removing the after_connect fails it) and is weak for Postgres, where a default server passes either way; noted as such at the test.
The expected probe vector was derived by reading the type map rather than by running it, and got the boolean arm wrong: sqlx reports a MySQL BOOLEAN column by its declared type, not as the TINYINT(1) it is stored as, so `trino_type` takes the BOOLEAN arm and the value renders as a real boolean. Both backends produce the same vector, which is the more useful thing to pin — a mapping moves between them unchanged — so it is now one shared constant.
The branch added docs/cli/sql.md without linking it, which docs_coverage::readme_indexes_every_command pins.
2 participants
Adds SQL graph sources: R2RML mappings over tables reached through any HTTP endpoint speaking the Trino client protocol — Trino / Starburst / PrestoDB directly, or the new
fluree-sql-bridgesidecar in front of a single Postgres, MySQL or SQLite database. No JDBC and no database drivers in theflureebinary; every scan is a statelessPOST /v1/statement+ paged GETs, so this works from a long-running server and from a Lambda alike.Design
The engine's R2RML boundary already asks providers for one table at a time (
scan_table(projection, filters, topk)), so a SQL source renders exactly oneSELECT … WHERE …per triples map and everything else — joins, OPTIONAL, UNION, aggregation — stays in-engine. No Ontop-style whole-query rewriting.SELECT * … LIMIT 0probe; an unsafe pairing (string vsbigint, tz-mismatched timestamp, NaN) is declined rather than guessed, since a mistyped comparison fails the whole statement in Trino and the in-engine FILTER stays authoritative regardless. Bound subjects push through template reversal;COUNTis answered by an exactSELECT COUNT(*) … IS NOT NULL(exact where Iceberg can only use manifest stats).LIMITslots for rows the mapping drops, breaking the superset contract.rr:sqlQueryis supported (SQL sources only): it compiles to a deterministic alias that stands in wherever a table name is expected, and scans as a derived table. Iceberg-backed registration refuses it up front.iceberg trackare refused with an error naming the source (full-rebuild materialization is a follow-up). Build watermarks recordsql://endpoint/table@time.endpointpolicy, not the catalog policy). The route is admin-token protected like/iceberg/map.Surface
fluree-db-sqlcrate (featuresql, on by default in server/CLI; stacks onicebergsince dispatch lives inFlureeR2rmlProviderand it reuses the sharedConfigValue/auth/secret-ref machinery)GraphSourceType::Sql/f:SqlMapping;Fluree::create_sql_graph_source;POST /sql/map;fluree sql map|list|info|drop(list/info/drop shared withfluree iceberg); hard drop sweeps the CAS mapping blob for all mapped-source familiesfluree-sql-bridge/: a standalone workspace (excluded liketestsuite-sparql, own lockfile and CI job) so sqlx and its three drivers never enter the main dependency treegraph-sources/sql.md,cli/sql.md, endpoints, vocabulary, crate map — plus a fix to the long-broken GRAPH-join example ingraph-sources/overview.md(object-with-"graph"-key syntax never parsed; the array form +from-namedis now shown and test-pinned)Verification
WHERE,COUNT(*), dates, zoned timestamps — all through the public API. Confirmed on the raw wire thatdecimal(10, 2)(with space) and… UTCrenderings decode.NUMERIC,TIMESTAMPTZ(offset normalized to UTC),TEXT[].FLUREE_SQL_BRIDGE_URL), and skips loudly when unset.sql-bridgejob, covering the injection regression, theLIMIT 0probe's Trino type names and value round-trips. A guard test fails if CI ever stops supplying the URLs, so a skipped backend cannot read as a passing one.FROM NAMED+GRAPH) and JSON-LD (["graph", …]+"from-named").Known gaps / candidate follow-up issues (none filed yet)
/iceberg/r2rml/generateis Iceberg-only).{"secret_ref": …}throughPOST /sql/mapandfluree sql map; both store credentials as literals in the graph-source record today (documented, matches the Iceberg REST catalog).fromNameddataset-local aliases don't resolve to graph sources, and the Iceberg path re-runs its nameservice lookup per scan (same memoization opportunity as fixed here for dispatch).Happy to file any of these if you agree they're worth tracking.
Review follow-ups (@aaj3f)
NO_BACKSLASH_ESCAPESon every MySQL session, and the renderer declines the pushdown for a MySQL string carrying a backslash rather than escaping for asql_modeit cannot observe —dialectnames the database behind an endpoint that need not be a bridge we configured. Declining is the same valve type mismatches already use, so it costs I/O and not correctness. Regression tests at both layers; the render-side test was verified to fail with the guard reverted.[lints] workspace = true: added. It was not clean underneath — three findings, fixed. The bridge is a separate workspace and cannot inherit, so it mirrors the table with a pointer to the root.row_countcost: documented next to theCOUNTclaim.standard_conforming_strings; the bridge now pins it per session as well.