feat(cli): fluree doc — Fluree Unstructured: documents into a searchable graph - #1760
Conversation
`f:queryVector` patterns only worked through `query_dataset_with_bm25`. The ordinary view path (`fluree.query(&view, …)`, and the two dataset paths in `dataset_query.rs`) wired `FlureeIndexProvider` as the BM25 provider but never as the vector provider, so every embedded vector search from the CLI or a plain `GraphDb` failed with "VectorSearch requires ExecutionContext.vector_provider". Wiring it exposed a second gap: the vector operator passes the view's `t` as `as_of_t` for any non-dataset query, and the embedded provider rejected any `as_of_t` at all as time travel. A head query on a single-ledger view therefore could never succeed. The guard now compares the requested `t` against the index watermark (`GraphSourceRecord:: index_t`): only a request for a state older than the snapshot is rejected, and the message says both numbers. Both changes sit behind `cfg(feature = "vector")`; without the feature nothing changes.
New crate `fluree-db-doc` plus `fluree doc ingest` / `fluree doc search`. `ingest` walks files or folders (PDF, Markdown, HTML, DOCX, PPTX, images) and writes one commit per document: the DoCO structure graph from the fluree-doc-parse engine (pinned by git rev), retrieval chunks cut along that structure with section paths and `doc:sourceElement` citations, an embedding per chunk when `[doc.embedding]` is configured, and a `doc:SourceDocument` node recording the file hash, parser revision and embedding model. It then creates or syncs a BM25 index (`<ledger>-text`) and, when vectors were produced, an HNSW index (`<ledger>-vectors`). Re-runs are cheap and idempotent: a document whose bytes, parser and embedding model are unchanged is skipped; a changed one is retracted by its `doc:sourceDocument` stamp and re-inserted, never diffed. A parse cache keyed on content hash and settings, and a reading cache keyed on crop pixels and model, sit under `.fluree/cache/doc/`. Model endpoints are three optional OpenAI-compatible slots in the config file's `[doc]` table — `embedding`, `llm`, and `vlm` (falls back to `llm`) — so OpenAI, Ollama, vLLM and the Fluree AI gateway configure the same way. Pages the deterministic parser cannot read are rendered as crops and sent to the `vlm` slot, spliced back under the engine's arbitration rules; with nothing configured the command makes no network connection. `search` embeds the query with the same endpoint (or runs BM25 with `--mode text`) and joins each hit to its chunk text, section path, file and chunk IRI. The CLI now builds with the `vector` feature by default so the HNSW index is available in-process.
`[doc.vlm]` / `[doc.llm]` gain `api = "chat" | "responses"`. The Fluree AI
gateway serves `/v1/responses` rather than chat completions and routes
the `doc-parse` intent to whichever vision provider the account holds
keys for, so with `api = "responses"` and `model = "auto"` the reader
sends `input_image` parts plus `"fluree": {"intent": "doc-parse"}` and
leaves the model choice to the gateway. A reading whose envelope is not
`completed` is refused rather than spliced as if whole.
Chat completions remain the default and are unchanged.
…AI login A configured CLI remote that points at a Fluree AI stack already holds the gateway URL and an OIDC login. `[doc] remote = "testco"` now supplies every model slot not set explicitly: embeddings against the gateway's `/v1/embeddings` (default `text-embedding-3-small`, which it forwards to the account's OpenAI-type provider), and crop reading plus extraction against `/v1/responses` with `model = "auto"`, letting the gateway route the intent to the account's vision provider. Explicit per-slot values win. The remote is looked up in the project config first and the global one second, so one `fluree auth login` from the home directory serves every project. A cheap authenticated call runs first so an expired login is refreshed and persisted before the token is copied into the slots. `RemoteLedgerClient` gains `current_token()` and `base_url()` for this. The cache tests name their temp directories by a counter instead of the wall clock, which collided under parallel runs.
The crop reader used reqwest's blocking client, which owns a runtime of
its own and panics when dropped inside the tokio context the command
runs in ("Cannot drop a runtime in a context where blocking is not
allowed"). ureq is a plain synchronous client with no runtime — the same
choice fdoc makes — and the reader runs inside the synchronous parse
anyway.
`read_doc_config` swallowed parse errors and reported the pipeline as
unconfigured, so a typo in `[doc]` ran a silent deterministic-only ingest
that looked like success. An absent section still means unconfigured; a
present but malformed one now fails with the file and the parse error.
Docs: the `doc.remote` setting and the account flow.
…tform config dirs `fluree remote add` run from the home directory writes to `~/.fluree`, which the CLI discovers by walking up rather than as the platform global directory, so the remote fallback missed it. Look there second.
A full-page reading of a page with no deterministic elements — a scan — was dropped by the engine's page replacement (fixed upstream in fluree/fluree-doc-parse@407daa0). The parse-cache fingerprint carries the revision, so cached scans re-parse once.
…again Every backend preserved a `retracted` flag when a graph source's config was re-published, so `bm25 drop` followed by `bm25 create` under the same name (and the vector and Iceberg/R2RML equivalents) reported success and left the record retracted: the new index existed in storage but no lookup or listing could see it. The create paths explicitly permit recreating a retracted source, so the two halves contradicted each other. `publish_graph_source` is only ever called from create and reconfigure flows — never from sync, which advances the index pointer through `publish_graph_source_index` — so publishing config now means the record is active. The index pointer is still preserved; retraction remains `retract_graph_source`'s to set. Applied to the file, memory, storage, DynamoDB and raft backends, with regression tests on the file and memory ones and the raft state-machine test updated to the new expectation.
…or index whose width changed A document routing more crops than `--max-crops` no longer fails the ingest. Nothing has been spent at that point, so the deterministic tier stands in, the document lands, the line says why, and the summary counts it; raising the cap re-parses it because the cap is part of the parse cache fingerprint. Switching embedding models changes the vector width, and vectors of a new size cannot be synced into an index built for the old one. The vector index is now dropped and rebuilt when the published dimensions differ from the embeddings just produced — which is what surfaced the nameservice bug fixed in the previous commit.
… calls made, publishing What one run builds, the three tiers (local only, Fluree AI as model gateway, hosted extraction as planned), registering and logging the CLI in to a Fluree AI stack, where the ledger lives and which gateway calls the vision, embedding and extraction steps make, how to publish the result to Fluree AI and what the indexes need there, and a troubleshooting table. Linked from the command reference, the guides index and the book.
… also reads ~/.fluree Verified against a Fluree AI stack: `fluree publish` created the hosted ledger and pushed every commit, chunks and embeddings included, but only once the remote was registered in the project's own config — the sync commands do not consult ~/.fluree the way `doc.remote` does.
… `fluree doc` docs/unstructured/ mirrors the Memory section's shape so the website can publish it as its own product: a landing page with the pitch and design philosophy; getting started (install, quickstart, connecting a Fluree AI account); concepts (what gets built and its vocabulary, the tiers, the three model slots and which calls each step makes and where data lives, re-runs and the two caches); guides (local models, publishing a built ledger to Fluree AI and what the indexes need there, querying the graph from a hit to the page, troubleshooting); the two subcommands; and a configuration and vocabulary reference. The section is self-contained — links do not leave the subtree, since the site drops cross-product links to plain text. The earlier standalone guide folds into it; the command reference in docs/cli/doc.md stays the flag-level page and points at the section, as docs/cli/memory.md does for Memory. Registered as a top-level part in SUMMARY.md, linked from the guides index, and announced next to Fluree Memory in the root README.
The landing page carries the opendataloader-bench standings the engine publishes (top 8 of 17, measured 2026-08-01, with the harness and metrics named and the caveats linked), mapped onto the tiers: the deterministic engine is the local tier and places third; the cascade with a vision model is a `vlm` slot or a Fluree AI account and places first. The tiers page carries the per-tier reading quality and cost.
The deterministic engine is the best model-free engine on the benchmark and ahead of most model-assisted ones; the vision-assisted cascade is ahead of everything measured on every metric. Stated on the landing page against the standings table, and in the root README blurb.
The standings table already carried them; the prose now does too, and the rest of the benchmark's public leaderboard — Unstructured, EdgeParse, MinerU, PyMuPDF4LLM, MarkItDown, LiteParse — is named with its source.
bplatz
added
enhancement
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz -- this is really neat, given what fluree-db-doc can do. I hadn't thought to shift this capability out of Fluree AI and into CLI-native capability. I think (as a human) the only things I want to surface because they're significant re: design choices & impact is (1) the move of vector as a feature flag into a default included with CLI (and, as I'm understanding it, a mandatory one). Some notes on the significance of that below that are worth just getting some explicit agreement for (not just Claude agreement) and then possibly some additional handling on the consequences of that and (2) not nearly as big, but it does seem we now have two HTTP client dependencies we're baking in: ureq alongside reqwest. Your commits and comments clarify the why, but just want to make sure we accept that we're proceeding anyway.
Claude review below:
This is a lot of careful work, @bplatz, and the crate boundary is exactly right: fluree-db-doc parses, escalates, chunks, embeds and emits a graph without knowing what a ledger is, the CLI owns writes, indexes and config, and the three tiers mean it works with nothing configured — which is also what CI exercises (the three Markdown integration tests run with no model). I verified rather than read: cargo test -p fluree-db-doc is 20/20, cargo clippy -p fluree-db-doc --all-targets -- -D warnings is clean, and cargo test -p fluree-db-nameservice is 165/165 with the two new recreate-after-retract tests. The two engine fixes riding along are both real — embedded vector search from a plain ledger view genuinely could not have returned rows before provider.rs:536 (every as_of_t was rejected as time travel), and bm25 drop then create really did publish an invisible index — and both have regression tests you verified red-first.
Two things before merge. The branch conflicts with main in Cargo.lock and fluree-db-cli/Cargo.toml (git merge-tree shows both), so it needs a rebase, and #1764 rides along. And the vector CLI feature needs one of two honest shapes: right now it's in default and commands/doc.rs calls the vector API with no cfg gate, so --no-default-features doesn't compile (five errors) and every cargo install and release target needs a C++ toolchain for usearch/cxx with no opt-out — it's what keeps --all-features from building on my machine. Either gate the doc vector paths and refuse cleanly when compiled out, or make it a plain dependency and say so in the release notes. Minor in code, but I'd rather it be a decision than the first failed cargo install.
Adherence to repo commitments:
- Patterns/abstractions: ✔ new crate with
[lints] workspace = true, workspace version/license; extendsGraphSourcePublishersemantics consistently across all four backends;FlureeIndexProviderreused as the vector provider rather than a new one. - Performance (speed first, memory second): ✔ provider pointer set at context build; one comparison per vector search; no engine hot-path change.
- Testing: ✔ 20 unit + 3 CLI integration (local, no network) in CI's nextest; nameservice regressions ×2 pinned;
⚠️ the embedding/vision network paths are tested only at the response-parsing layer (reasonable for a CLI, worth saying). - Conventions: ✔ thorough multi-line commits; docs tree registered in
SUMMARY.md;DOC_PARSE_REVpin note;⚠️ branch conflicts withmain.
Verified locally at branch HEAD 05cac473a: cargo test -p fluree-db-doc → 20/20; cargo clippy -p fluree-db-doc --all-targets -- -D warnings → clean; cargo test -p fluree-db-nameservice → 165/165; git merge-tree origin/main HEAD → 2 conflicts; cargo clippy -p fluree-db-cli --no-default-features --features server,iceberg,shacl,aws → 5 compile errors (vector API used unconditionally); fluree/fluree-doc-parse confirmed public.
Approving so you can merge once it's rebased — and let's settle the vector default first.
| # connection config. Dormant unless configured; ~+2 MB over the AWS SDK that | ||
| # `iceberg` already pulls in. | ||
| default = ["server", "iceberg", "shacl", "aws"] | ||
| default = ["server", "iceberg", "shacl", "aws", "vector"] |
There was a problem hiding this comment.
Should-fix (fold in now). default = [..., "vector"] makes usearch → cxx part of every default CLI build — and as written the feature can't be turned off: commands/doc.rs calls create_vector_index / sync_vector_index / drop_vector_index / VectorCreateConfig with no #[cfg(feature = "vector")] gate anywhere (none in doc.rs or index.rs; the CLI had no such calls at BASE).
Concretely, cargo build -p fluree-db-cli --no-default-features --features server,iceberg,shacl,aws fails with five E0599/E0433 errors (I hit them on the #1764 head, same code).
That matters because cxx needs a C++ toolchain at build time — it's exactly what stops --all-features building on my Mac — so every cargo install fluree-db-cli and every release target now needs one, with no opt-out.
Two honest shapes: gate the doc vector paths on the feature and have doc search --mode vector refuse with a clear message when compiled out, or drop the pretense and make vector a plain dependency with a release-notes line. Either is fine; a default feature that cannot be disabled is the one shape I'd not ship.
| QueryError::InvalidQuery(format!("Graph source not found: {graph_source_id}")) | ||
| })?; | ||
|
|
||
| // Vector indexes are head-only. A single-ledger view always names its |
There was a problem hiding this comment.
Praise. This is the right fix for a real bug: at BASE every as_of_t was rejected as time travel, but a single-ledger view always names its t, so embedded vector search from a plain view could never return rows. Rejecting only t < record.index_t and serving the head index otherwise ("possibly stale, exactly as for a head query") is the monotone semantics I'd want, and the error message naming both ts will save someone an afternoon.
| } | ||
| None => "ready".to_string(), | ||
| }; | ||
| // Publishing config is what creating or reconfiguring a graph |
There was a problem hiding this comment.
Praise, and one note. Clearing retracted on publish_graph_source across all four backends (with the DynamoDB if_not_exists removed at dynamodb/mod.rs:1908-1916) is the correct meaning of "publish config = create or reconfigure", and the two regression tests pin it.
Worth a line in the description for fluree/solo's benefit: this is a behavior change on a shared trait, not a signature change, so solo picks it up silently on its next repin — which is what its virtual-dataset path wants, but it should know.
| fluree-doc-docx.workspace = true | ||
| fluree-doc-html.workspace = true | ||
| fluree-doc-pptx.workspace = true | ||
| hayro-syntax.workspace = true |
There was a problem hiding this comment.
Optional. ureq alongside reqwest is well justified in the comment (blocking client inside the synchronous parse; reqwest's blocking client panics when dropped inside tokio). It is a second HTTP stack in the tree, though; if the parse ever becomes async, worth collapsing.
| ## Three ways to run it | ||
|
|
||
| | Tier | What you need | What you get | | ||
| |---|---|---| |
There was a problem hiding this comment.
Optional (docs). If vector stays a default feature, the install story should say a C++ toolchain is required to build the CLI; if it becomes optional, that the deterministic tier needs none. Ties to the Cargo.toml note.
# Conflicts: # Cargo.lock # fluree-db-cli/Cargo.toml
`fluree doc` required usearch: `commands/doc.rs` called the embedded HNSW lifecycle (`create`/`sync`/`drop_vector_index`) with no `cfg` gate, so the declared `vector` feature could not be turned off — building the CLI with `--no-default-features` failed with five errors, and every `cargo install` and release target needed a C++ toolchain. The premise was wrong. `vector` gates one *backend*, not the capability: `fluree-db-query/src/` has exactly one such gate (`vector/mod.rs`, the usearch module). Flatrank — `cosineSimilarity`/`dotProduct` over `@vector` literals — is always compiled. So `doc search --mode vector` can rank chunks with no index and no ANN library at all. `doc ingest` no longer builds a vector graph source, and `doc search` scores every chunk's `doc:embedding` with `cosineSimilarity`, ordered and cut by LIMIT. Embeddings are still written: they are ledger data, so they time-travel, need no sync, and survive an embedding-model change without an index to rebuild for the new width. Cost is linear in chunks, which is the right trade for a folder of documents and the reason the approximate index belongs in `fluree server` (built with its `vector` feature) rather than in a local CLI. The `vector` feature is dropped from `fluree-db-cli` entirely rather than gated: CI builds `--all-features` everywhere, so a cfg'd-out configuration would be one nothing ever compiles. Test: a stub `/embeddings` endpoint gives each chunk a marker-word vector, then two opposite queries must invert the ranking — an unscored scan cannot pass both. Verified non-vacuous by flipping the sort to `asc`, which fails it (0.997 vs 0.003, real cosine separation).
|
Thanks — all addressed, and the
That closes your build objection at the root: On Good call on the nameservice note — it's in the description now, flagged for solo's next repin. Branch is merged up with Hybrid search and the extraction stage are in the follow-up PR stacked on this one; testing for those lands there. |
#1760 landed on main, so the vector lane's HNSW index is gone. Conflicts resolved by keeping this branch's structure — two concurrent lanes, hybrid fusion, the post-ingest ledger index — and swapping what the vector lane runs. The lane now scores every chunk's `doc:embedding` with `cosineSimilarity`, ordered and cut by LIMIT, instead of searching a graph source. Fusion is untouched: it calibrates against a cosine in 0..1, which is exactly what flatrank produces, so the same scores arrive on the same scale. `search_hits` takes the where-clauses that bind `?c` and `?score` plus an optional query vector, rather than a single index pattern, so both lanes build their own head and share the chunk join. Mode selection asks whether the chunks carry embeddings (`has_embeddings`) rather than whether a vector graph source exists, and the "no vector index" error becomes one about embeddings, since that is now the real precondition.
2 participants
What this adds
fluree doc ingest <folder>turns PDFs, DOCX, PPTX, Markdown, HTML and scans into a ledger holding, per document, the DoCO structure graph fromfluree-doc-parse, retrieval chunks cut along that structure (each citing its source elements and carrying its section path), an embedding per chunk, and a document node with hash, parser revision and embedding model. It then creates or syncs a BM25 index over the chunks.fluree doc searchsearches by words (BM25) or by meaning (cosine similarity over the embeddings) and joins each hit back to chunk text, section path and file.Three tiers, same pipeline:
[doc.embedding],[doc.vlm],[doc.llm]slots, any OpenAI-compatible endpoint (Ollama, vLLM, OpenAI…).vlmreads crops of pages the parser could not; falls back tollm.doc.remote = "<remote>"fills unset slots from the remote's gateway and stored OIDC login. The ledger stays local; only model calls go to the account.Re-runs skip unchanged documents (same bytes, parser revision, embedding model), retract-and-replace changed ones, and cache parses on content and vision readings on crop pixels. A document over
--max-cropslands deterministic-only with a note.New crate
fluree-db-docowns the document side (parse, escalation, chunk, embed, caches, graph emission); the CLI owns ledger writes, indexes and config.Vector search without an ANN library
--mode vectorscores every chunk'sdoc:embeddingwithcosineSimilarity, ordered and cut byLIMIT. There is no HNSW index and the CLI links no vector library.This started as review feedback — the CLI declared a
vectorfeature but called the embedded index API ungated, so--no-default-featuresfailed to build and everycargo installneeded a C++ toolchain forusearch. The premise underneath was wrong:vectorgates one backend, not the capability.fluree-db-query/src/has exactly one such gate (vector/mod.rs, the usearch module); the IR pattern, the operator, the provider trait and the flatrank functions are always compiled. Ranking chunks needs none of it.So the feature is dropped from
fluree-db-clientirely rather thancfg-gated — CI builds--all-featureseverywhere, so a gated-out configuration would be one nothing ever compiles. What that buys:--atworks — a vector search is an ordinary query over an ordinary property.Cost is linear in chunks, which is the right trade for a folder and the reason the approximate index belongs in
fluree server(built with its ownvectorfeature) rather than in a local CLI.--mode textstays indexed at any size.docs/unstructured/concepts/vector-search.mddocuments the boundary.Follow-up: a
vectorfeature and/vector/create+/vector/syncroutes on the server, andfluree docopting into the existing local-server auto-routing (context.rs:675), so a corpus that outgrows a scan gets an HNSW index from a running server. None of that exists today — the server has novectorfeature and no vector routes.Bugs fixed on the way (separate commits)
retractedflag, sobm25 dropthenbm25 createunder the same name reported success and produced an invisible index. Publishing config now means active; the raft test's expectation is updated and regression tests added for file and memory backends. Note for fluree/solo: this is a semantic change on a shared trait, not a signature change, so solo picks it up silently on its next repin. It is what its virtual-dataset path wants, but it should know.Two HTTP clients, deliberately
fluree-db-docpullsureqalongsidereqwest. Crop reading runs inside the synchronous parse (escalate.rs), andreqwest's blocking client owns a runtime that panics when dropped inside tokio;ureqis a plain blocking client with no runtime of its own. The alternatives were a channel back to the async runtime or restructuring the parse, both worse than one small pure-Rust client. Accepting the second stack knowingly — if the parse ever goes async, it collapses back to one.Docs
docs/unstructured/is a product-scoped section in the Memory section's shape (landing with benchmark standings, getting started, concepts, guides, CLI, reference), registered inSUMMARY.mdand linked from the guides index and the root README.docs/cli/doc.mdis the flag-level reference. The website repo has the matching product sync (make sync-unstructured).Verified
fluree-db-doc(20), CLI integration tests for ingest/search/skip/force/dry-run (4, one new), docs coverage, nameservice (165), consensus state-machine, API graph-source suites./embeddingsendpoint: two opposite queries must invert the ranking, so an unscored scan cannot pass. Shown non-vacuous by flipping the sort toasc, which fails it (0.997 vs 0.003, real cosine separation). The stub proves the query shape and ordering; a run against a real embedding model over a real corpus is still worth doing before merge.cargo build -p fluree-db-cli --no-default-features --features server,iceberg,shacl,awsnow succeeds (five errors at review time).doc.remoteset: a scanned PDF read through the gateway's proxy to Gemini, spliced, chunked and embedded at 1536 dims through the proxy, then returned as the top vector hit;fluree publishmoved the ledger to the stack with all chunks and embeddings. This run predates the flatrank change and exercised the HNSW path, so it stands as evidence for the parse/escalate/embed/publish pipeline but not for the current search lane.Not in this PR
Entity and relation extraction (
--model/--entities, thellmslot), a server route for ingest, server-side HNSW indexes and the routing to reach them, and handing a folder to Fluree AI's hosted extraction.https://claude.ai/code/session_01AiGLEyFBDSXiUdYBh2RhX6