Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,65 @@ jobs:
timeout-minutes: 45
run: cargo test

sql-bridge:
# fluree-sql-bridge is a standalone workspace (it links sqlx + three
# database drivers, none of which belong in the fluree binary), so the
# workspace gates above never reach it.
runs-on: ubuntu-latest
# SQLite exercises the protocol in-process, but it is the one backend whose
# string literals cannot misbehave. The escaping rule the bridge enforces
# (NO_BACKSLASH_ESCAPES on MySQL) is only observable against a real server,
# so both are supplied here. `server_backends_are_configured_in_ci` fails if
# these ever stop being set, so a skipped test cannot read as a pass.
services:
mysql:
image: mysql:8
env:
MYSQL_ROOT_PASSWORD: fluree
MYSQL_DATABASE: bridge_test
ports:
- 3306:3306
options: >-
--health-cmd="mysqladmin ping -h 127.0.0.1 -uroot -pfluree"
--health-interval=5s
--health-timeout=5s
--health-retries=30
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: fluree
POSTGRES_DB: bridge_test
ports:
- 5432:5432
options: >-
--health-cmd=pg_isready
--health-interval=5s
--health-timeout=5s
--health-retries=30
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@1.97.0
- uses: rui314/setup-mold@v1
- uses: Swatinem/rust-cache@v2
with:
cache-on-failure: true
workspaces: "fluree-sql-bridge -> target"

- name: Format
working-directory: fluree-sql-bridge
run: cargo fmt --all -- --check

- name: Clippy
working-directory: fluree-sql-bridge
run: cargo clippy --all-targets -- -D warnings

- name: Test (SQLite, MySQL and Postgres backed protocol tests)
working-directory: fluree-sql-bridge
env:
FLUREE_BRIDGE_MYSQL_URL: mysql://root:fluree@127.0.0.1:3306/bridge_test
FLUREE_BRIDGE_POSTGRES_URL: postgres://postgres:fluree@127.0.0.1:5432/bridge_test
run: cargo test

bench-paths:
# Cost gate for bench-compare below, which is a ~30-minute job. GitHub's
# `paths:` filter is workflow-scoped and the rest of this workflow must run
Expand Down
22 changes: 22 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ members = [
"fluree-db-iceberg",
"fluree-db-tabular",
"fluree-db-r2rml",
"fluree-db-sql",
"fluree-db-server",
"fluree-db-bolt",
"fluree-db-peer",
Expand All @@ -48,7 +49,7 @@ members = [
"fluree-db-consensus",
"fluree-raft-core",
]
exclude = ["testsuite-sparql", "testsuite-shacl", "scripts/local/load"]
exclude = ["testsuite-sparql", "testsuite-shacl", "scripts/local/load", "fluree-sql-bridge"]

[workspace.package]
version = "4.1.6"
Expand Down
2 changes: 2 additions & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
- [mcp](cli/mcp.md)
- [docs](cli/docs.md)
- [iceberg](cli/iceberg.md)
- [sql](cli/sql.md)
- [bm25](cli/bm25.md)
- [materialize](cli/materialize.md)
- [completions](cli/completions.md)
Expand Down Expand Up @@ -174,6 +175,7 @@
- [Overview](graph-sources/overview.md)
- [Iceberg / Parquet](graph-sources/iceberg.md)
- [R2RML](graph-sources/r2rml.md)
- [SQL endpoints (Trino / bridge)](graph-sources/sql.md)
- [BM25 graph source](graph-sources/bm25.md)

- [Fluree for AI and agents](ai/README.md)
Expand Down
69 changes: 68 additions & 1 deletion docs/api/endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -1750,7 +1750,7 @@ A flat array of ledgers and graph sources. Retracted entries are omitted.
{"name": "mydb", "branch": "dev", "type": "Ledger", "t": 3},
{"name": "docsearch", "branch": "main", "type": "BM25", "t": 5,
"dependencies": ["mydb:main"]},
{"name": "warehouse", "branch": "main", "type": "Iceberg", "t": 0,
{"name": "orders-db", "branch": "main", "type": "Iceberg", "t": 0,
"dependencies": ["mydb:main"]}
]
```
Expand Down Expand Up @@ -3042,6 +3042,73 @@ By default the server does not sync on commit, so an index only advances when so

See also the CLI equivalent: [fluree bm25 sync](../cli/bm25.md#fluree-bm25-sync).

### POST {api_base_url}/sql/map

Map tables behind a SQL endpoint as an R2RML graph source. The endpoint speaks the Trino client protocol (Trino, Starburst, PrestoDB, or a `fluree-sql-bridge` sidecar). Admin-protected — requires the admin Bearer token when an admin token is configured. Available only when the server is built with the `sql` feature (on by default). See [SQL graph sources](../graph-sources/sql.md).

**URL:**
```
POST {api_base_url}/sql/map
```

**Request Body:**

```json
{
"name": "orders-db",
"endpoint": "https://trino.example.com:8443",
"r2rml": "@prefix rr: <http://www.w3.org/ns/r2rml#> . ...",
"r2rml_type": "text/turtle",
"branch": "main",
"dialect": "trino",
"protocol": "trino",
"catalog": "hive",
"schema": "sales",
"user": "fluree",
"auth_bearer": "…",
"session": { "query_max_run_time": "5m" }
}
```

| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Graph source name (required) |
| `endpoint` | string | Statement endpoint base URL (required); `/v1/statement` is appended. Loopback/private hosts are allowed; the link-local/metadata range is refused. |
| `r2rml` | string | Inline R2RML mapping (required). `rr:tableName` and `rr:sqlQuery` logical tables are both accepted. |
| `r2rml_type` | string | Media type of `r2rml` (`text/turtle`, `application/ld+json`) |
| `branch` | string | Branch name (default: `main`) |
| `dialect` | string | `trino` (default), `postgres`, `mysql`, `sqlite` — the engine behind a bridge |
| `protocol` | string | `trino` (default, `X-Trino-*` headers) or `presto` |
| `catalog`, `schema` | string | Defaults for unqualified table names |
| `user` | string | Protocol user header (default `fluree`) |
| `auth_bearer` | string | Static bearer token |
| `oauth2_token_url`, `oauth2_client_id`, `oauth2_client_secret`, `oauth2_scope`, `oauth2_audience` | string | OAuth2 client-credentials flow (refreshes); `oauth2_token_url` is guarded against internal hosts |
| `session` | object | Session properties sent as `X-Trino-Session` |

**Response:**

```json
{
"graph_source_id": "orders-db:main",
"endpoint": "https://trino.example.com:8443",
"connection_tested": true,
"mapping_source": "bafy…",
"triples_map_count": 3,
"table_count": 2,
"table_names": ["sales.customers", "sales.orders"],
"mapping_validated": true
}
```

`connection_tested` reports whether `SELECT 1` succeeded against the endpoint; a failure does not block registration.

**Status Codes:**
- `201 Created` — graph source created
- `400 Bad Request` — invalid body, unknown `dialect`/`protocol`, endpoint refused by the SSRF guard, or an invalid mapping
- `401 Unauthorized` — admin token required

---

### POST {api_base_url}/iceberg/materialize

Materialize a graph source into a native ledger (so BM25 / vector / reasoning can run over it). Reads incrementally from a per-`(source, target, table)` watermark persisted in a shared `fluree_materialize_state:main` ledger, or fully with `force_full`. `target` may be a template that fans out into one ledger per partition (see the field table). Admin-protected; `iceberg` feature only. See [Materialization](../graph-sources/iceberg.md#materialization-into-a-native-ledger).
Expand Down
1 change: 1 addition & 0 deletions docs/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ fluree query 'SELECT ?name WHERE { ?s <http://example.org/name> ?name }'
| [`reindex`](reindex.md) | Full reindex from commit history |
| [`sweep`](sweep.md) | Reclaim index artifacts no index chain references |
| [`iceberg`](iceberg.md) | Map and manage Iceberg tables as graph sources (map, list, info, drop) |
| [`sql`](sql.md) | Map and manage SQL tables as graph sources through a Trino-protocol endpoint (map, list, info, drop) |
| [`materialize`](materialize.md) | Build a native ledger twin from a virtual (Iceberg/R2RML) graph source |
| [`bm25`](bm25.md) | Manage BM25 full-text search indexes (create, list, sync, drop) |

Expand Down
113 changes: 113 additions & 0 deletions docs/cli/sql.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# fluree sql

Manage SQL graph sources — R2RML mappings over tables reached through a
Trino-protocol endpoint (Trino, Starburst, PrestoDB, or a `fluree-sql-bridge`
sidecar). See [SQL graph sources](../graph-sources/sql.md).

## Subcommands

| Subcommand | Description |
|------------|-------------|
| `map` | Map tables behind a SQL endpoint as a graph source |
| `list` | List mapped graph sources (SQL, Iceberg and R2RML) |
| `info` | Show details for a mapped graph source |
| `drop` | Drop a mapped graph source |

`list`, `info` and `drop` are shared with [`fluree iceberg`](iceberg.md): both
commands operate on the same family of mapped sources.

## fluree sql map

### Usage

```bash
fluree sql map <NAME> --endpoint <URL> --r2rml <PATH> [OPTIONS]
```

### Arguments

| Argument | Description |
|----------|-------------|
| `<NAME>` | Graph source name (e.g., "orders-db") |

### Options

**Endpoint:**

| Option | Description |
|--------|-------------|
| `--endpoint <URL>` | Statement endpoint base URL (required), e.g. `https://trino.example.com:8443` or `http://localhost:8080` for a sidecar |
| `--dialect <NAME>` | SQL rendering dialect: `trino` (default), `postgres`, `mysql`, `sqlite`. Use the engine behind a bridge. |
| `--protocol <NAME>` | Header family: `trino` (default) or `presto` |
| `--catalog <NAME>` | Default catalog for unqualified table names |
| `--schema <NAME>` | Default schema for unqualified table names |
| `--user <NAME>` | Protocol user (`X-Trino-User`); defaults to `fluree` |
| `--session KEY=VALUE` | Session property (repeatable), e.g. `--session query_max_run_time=5m` |

**R2RML mapping:**

| Option | Description |
|--------|-------------|
| `--r2rml <PATH>` | Mapping file (required). Each `rr:tableName` names a table reachable through the endpoint; `rr:sqlQuery` is also accepted. |
| `--r2rml-type <TYPE>` | Mapping media type (e.g., `text/turtle`); inferred from extension if omitted |

**Authentication:**

| Option | Description |
|--------|-------------|
| `--auth-bearer <TOKEN>` | Static bearer token |
| `--oauth2-token-url <URL>` | OAuth2 client-credentials token endpoint |
| `--oauth2-client-id <ID>` | OAuth2 client ID |
| `--oauth2-client-secret <SECRET>` | OAuth2 client secret |
| `--oauth2-scope <SCOPE>` | OAuth2 scope |
| `--oauth2-audience <AUD>` | OAuth2 audience |

**General:**

| Option | Description |
|--------|-------------|
| `--branch <BRANCH>` | Branch name (defaults to `main`) |
| `--remote <NAME>` | Execute against a remote server |

### Examples

```bash
# Trino with a bearer token; tables are qualified inside hive.sales
fluree sql map orders-db \
--endpoint https://trino.example.com:8443 \
--catalog hive --schema sales \
--auth-bearer "$TRINO_TOKEN" \
--r2rml mappings/orders.ttl

# A bridge sidecar in front of Postgres
fluree sql map crm \
--endpoint http://localhost:8080 \
--dialect postgres --schema public \
--r2rml mappings/crm.ttl
```

### Output

```
Mapped SQL endpoint as graph source 'orders-db:main'
Endpoint: https://trino.example.com:8443
R2RML: bafy…
TriplesMaps: 3
Tables: 2 (sales.orders, sales.customers)
Connection: verified
Mapping: validated
```

`Connection: not tested` means the `SELECT 1` probe failed; the source is
still registered and the first query reports the underlying error.

## fluree sql list / info / drop

```bash
fluree sql list
fluree sql info orders-db
fluree sql drop orders-db --force
```

Behave exactly as the [`fluree iceberg`](iceberg.md) equivalents; SQL sources
show the type `SQL`.
16 changes: 16 additions & 0 deletions docs/concepts/graph-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,22 @@ WHERE {

See the [R2RML documentation](../graph-sources/r2rml.md) for details.

### SQL Endpoints

**Differentiator**: The R2RML mapping runs over a live relational database or warehouse through a Trino-protocol HTTP endpoint — no copy, and no database driver inside Fluree. One Trino coordinator reaches Postgres, MySQL, SQL Server, Oracle, Snowflake, BigQuery and more; a small `fluree-sql-bridge` sidecar covers a single Postgres/MySQL/SQLite database without a JVM.

**Use Cases:**
- A virtual graph over an operational database
- Federating a ledger with warehouse tables in one query
- Serverless deployments — every scan is a stateless HTTP request

**Key Features:**
- Typed filter pushdown and exact `COUNT` per table; joins in the engine
- `rr:sqlQuery` logical tables
- Reads the current table state (no snapshots or time travel)

See [SQL graph sources](../graph-sources/sql.md) for details.

## Graph Source Lifecycle

### Creation
Expand Down
Loading
Loading