fix(ingest): store an object-position empty collection as the rdf:nil triple it denotes

[aaj3f]

Partially addresses #1694 — this takes the silent statement loss half only; the D-13 decision about how non-empty collection structure is stored (materialize the rdf:first/rdf:rest spine vs documented divergence) stays open and this PR deliberately does not touch it.

The bug — one silent-loss class, two ingest surfaces

An object-position empty collection denotes the IRI rdf:nil — ex:s2 ex:items () IS ex:s2 ex:items rdf:nil, and {"ex:items": {"@list": []}} is the same statement spoken in JSON-LD. Both ingest surfaces lost it silently:

Turtle. The default CollectionStyle::IndexedItems path had a Nil arm in parse_object_list (fluree-graph-turtle/src/parser.rs) that advanced the lexer and emitted nothing: the statement was consumed, zero flakes were written, no diagnostic fired, and the subject simply never existed in the database. The issue's repro (an 8-triple document committing 3 flakes, ex:s2 unqueryable) reproduces exactly, end-to-end through insert_turtle — the e2e test failed at the merge base with ex:s2 must exist: [] before the fix went in. And the NIL token is '(' WS* ')' with comments not counting as WS, so ex:s ex:p ( # c\n ) . lexes as LParen/RParen instead and reached parse_collection_as_list with zero items — a second spelling of the same loss that bypasses the Nil token entirely (credit to @bplatz for catching it; the repro emits zero triples exactly as described).

JSON-LD. Three sites, same class. parse_list_values_with_ctx (fluree-db-transact/src/parse/jsonld.rs) produced zero triple templates for {"@list": []} — pinned, at the time, by test_parse_empty_list as intended behavior — and the singular fallback parse_list_value_with_ctx errored with Empty @list in unexpected position. The import-path adapter (fluree-graph-json-ld/src/adapter.rs, process_list, feeding bulk import and the inline shapes/ontology loaders) emitted zero events. And the expansion itself (fluree-graph-json-ld/src/expand.rs) drops any key whose values expand to nothing, @list included — so the explicit-array spelling [{"@list": []}] expanded to [{}] and stored a spurious blank node object: not even a loss, but wrong data.

The fix

Turtle: delete the special case. With the Nil arm gone, the token falls through to parse_object, which already resolves () to the rdf:nil term — the exact code path a literally-written rdf:nil object takes, in both collection styles. I verified the downstream premise first: an explicit ex:s2 ex:items rdf:nil . already stored and queried fine (it's just an IRI object; the control test pins it), so lowering () to that same path is provably consistent rather than a new behavior — one code path, indistinguishable output (default_empty_collection_equals_written_rdf_nil asserts the two graphs are identical). For the comment spelling, parse_collection_as_list now emits the rdf:nil triple when the item loop ran zero times and returns the term so parse_object_list can admit an RDF 1.2 annotation tail on it — making ( # c\n ) fully equivalent to (): same triple, same annotation admission, which is what keeps the CollectionStyle doc's "single term in both styles" claim true for every spelling. I weighed the fail-loudly alternative (refuse the statement) and I don't think it survives contact: it rejects valid Turtle and breaks previously-accepted documents at a release boundary, and there's nothing structural forcing us there — the nil triple just works.

JSON-LD: the same term at each site. parse_list_values_with_ctx yields one rdf:nil template (ordinary IRI object, no list_index), the singular fallback returns the same term instead of erroring, process_list emits the rdf:nil triple, and the expansion's drop-empty-keys rule now exempts @list so [{"@list": []}] expands to itself. Every spelling — bare object, explicit array position — now stores exactly the one triple, and it is byte-identical to what the Turtle surface stores (the parity e2e asserts the two ledgers' full ?s ?p ?o row sets are equal). SPARQL needed nothing: term.rs:642 already lowers a pattern-position () to the rdf:nil constant, which is exactly why basic#list-1 greens below.

On the hot path this is a deletion plus a zero-item branch, not per-item work — documents without collections see no new per-statement cost, and non-empty collections take the same arms they always did on both surfaces. default_mode_output_is_pinned_triple_for_triple pins the full default-mode Turtle surface triple-for-triple (now including the nil edge), and the e2e suites assert non-empty collections keep their list_index shape with no spine materialized (?l rdf:first ?x still matches nothing — that's D-13, not this PR).

Fallout taken deliberately

W3C basic#list-1 flips green and comes off the register. Its query (:x ?p ()) lowers to the rdf:nil constant on the SPARQL side (no spine needed), and its data (:x :list0 ()) now ingests as :x :list0 rdf:nil — so it passes outright. The both-way register check would have failed the run on the stale entry; list-1 is removed and the comment rewritten. list-2..4 genuinely need the first/rest spine and stay registered — they come off with D-13, whichever way it lands. Full suite is green: 36/36 with the updated register. The matching stale note in docs/audit/burn-down/ROADMAP.md (where D-13 is tracked) is corrected in the same spirit: ingest no longer drops () objects, list-1 greens, list-2..4 remain D-13's.

RDF 1.2 annotations on () now parse in the default mode. Previously :s :p () {| :q :z |} was refused alongside non-empty collections; with the base triple existing there's a term to reify, and refusing it while accepting the identical :s :p rdf:nil {| :q :z |} would be an inconsistency. This holds for the comment spelling too. Annotations on non-empty collections stay refused in IndexedItems (nothing to reify there) — that existing test is untouched.

Empty @list stops erroring in the singular position. The Empty @list in unexpected position error is gone; the fallback returns the term the value denotes, consistent with the array path. test_parse_empty_list is updated to pin the new single-rdf:nil-template behavior, with a new sibling pinning the singular fallback.

Subject position was already correct. () ex:p ex:o . has always parsed as an rdf:nil subject (parse_subject resolves the Nil token) — so it's not the same bug class, but it's now pinned at both the parser and e2e level so the two positions can't diverge again. Nested () inside a non-empty collection was also already handled (it arrives as an rdf:nil list item) and is now pinned too.

Verification

• Turtle e2e suite fluree-db-api/tests/it_turtle_empty_collection.rs (grp_transact): the issue's repro document, the literal-rdf:nil control (identical row sets), the SPARQL ()-pattern lookup finding ex:s2, no-spine-materialized, and subject-position (). Non-vacuity proven both directions: written first and failed at the merge base, and re-failed when I temporarily reinstated the deleted arm on the final tree.
• JSON-LD e2e suite fluree-db-api/tests/it_jsonld_empty_list.rs (grp_transact): the repro shape via {"@list": []}, the Turtle↔JSON-LD parity assertion (byte-identical graphs), and the array-position spelling pinned to rdf:nil-not-a-blank-node. All written first and failed before the fixes (the bare-object tests with ex:s2 must exist: []; the array-position test by storing a blank node).
• Parser/unit level: the stale pins (test_empty_collection, default_empty_object_collection_still_emits_nothing, transact test_parse_empty_list, adapter test_empty_list) updated to assert the nil triple; new pins for ()≡rdf:nil equivalence, the comment spelling (default_empty_collection_with_comment_is_rdf_nil), nested-() items, subject position, annotation behavior on both spellings, the singular @list fallback, and expansion keeping the @list key (test_expand_empty_list_keeps_list_key). Each of the three JSON-LD fixes was individually reverted to watch its test fail, then restored. Spine/conformant tests untouched and green — ParserOptions::conformant() behavior is unchanged.
• Gates: -p fluree-graph-turtle (139 lib + integration suites), -p fluree-graph-json-ld (143 lib + 12 + 23 + 8), -p fluree-db-transact (306 + 46), -p fluree-db-api grp_transact/grp_query/grp_query_sparql/grp_import/grp_misc/grp_policy/grp_graphsource (197/422/369/101/263/96/143), full W3C SPARQL suite 36/36 with the updated register, workspace cargo check --all-targets clean, per-crate clippy -D warnings clean on all four touched crates, fmt checked in both the root and testsuite-sparql workspaces.

What this closes and what it doesn't

Closed: an object-position empty collection can no longer vanish (or mint a spurious blank node) on either ingest surface. The Turtle parser path serves insert_turtle, bulk/chunked import of Turtle, and the fluree rdf tooling; the JSON-LD path is its own pipeline — insert/transact via the template parser, and import/inline-shapes/inline-ontology via expand + to_graph_events — and each of its three lossy sites is fixed and pinned. SPARQL pattern position needed no change (term.rs:642). Not closed: whether Fluree materializes the first/rest spine for non-empty collections (basic#list-2..4, round-trip fidelity) — that's the product call the issue tracks as D-13, and nothing here prejudices either answer. Also untouched, deliberately: << :s :p () >> inside an RDF 1.2 reified-triple term stays rejected — the RDF 1.2 grammar's rtObject excludes collections, so that's conformant behavior, not a gap.

[bplatz]

Approving — the approach is right (deleting the arm rather than adding one), and default_empty_collection_equals_written_rdf_nil is the pin that makes it provable rather than asserted.

Verified locally: -p fluree-graph-turtle green (137 lib + integration), the new e2e 4/4, sparql10_query_eval_tests green with the register change (the both-way check makes that real evidence list-1 passes and list-2..4 still don't), fmt clean in both workspaces, clippy clean.

Two things to look at before/after merge — see the inline comments, plus:

JSON-LD twin surface is untouched. {"@list": []} still produces zero triples (fluree-db-transact/src/parse/jsonld.rs, parse_list_values_with_ctx: "Empty list produces zero triples", pinned by test_parse_empty_list); in the non-array position it errors with Empty @list in unexpected position (should be handled by parse_expanded_objects). Per JSON-LD→RDF an empty @list is rdf:nil — same silent-loss class, same issue. Fix here or file a follow-up; either way the body's "every ingest surface goes through the one fixed parser path" should be scoped to Turtle, since JSON-LD ingest doesn't touch this parser. (SPARQL is already fine — term.rs:642 lowers () to the rdf:nil constant.)

Nit: << :s :p () >> is still rejected at parser.rs:1323 as "collections and blank-node property lists". Grammar-conformant (rtObject excludes collections), just slightly at odds with the "() is just an IRI" framing. Fine to leave.

Also checked and not a problem: mixing indexed items and a nil object on the same (s,p) doesn't create a new retraction list-index hydration hazard — it was already reachable by writing rdf:nil explicitly.

[bplatz]

() is still special-cased when it doesn't lex as Nil. Turtle's NIL is '(' WS* ')' and comments aren't WS, so ( # c\n ) lexes LParen/RParen, takes the LParen if indexed_lists arm, loops zero times in parse_collection_as_list, and emits nothing — the original bug, still silent. Confirmed on this branch:

ex:s ex:p ( # c\n ) .  →  0 triples
ex:s ex:p (\n) .       →  1 triple, rdf:nil

Spine mode handles it (parse_collection returns rdf_nil() at :1131), so the styles diverge here — which also makes the new options.rs:38-41 line ("() is a single term — rdf:nil — in both styles") not quite true.

Three lines in parse_collection_as_list close it; I ran this, 138/138 pass and the comment form yields the nil triple:

if index == 0 {
    let nil = p.rdf_nil();
    p.sink_emit_triple(subject, predicate, nil)?;
}
p.expect(&TokenKind::RParen)?;

Worth a test on the comment spelling, since it's the only input that reaches it.

[aaj3f]

You're right, and confirmed by execution before touching anything: I wrote the test first against this branch's HEAD and ex:s ex:p ( # nothing here\n ) . came back [] — zero triples, the original silent loss through the one spelling that bypasses the Nil token. Spine mode returns rdf_nil() for the same input, so the styles did diverge exactly as you said, and the options.rs "in both styles" line was overclaiming.

Fixed in 3e2a723 with your zero-item guard, plus one extension: parse_collection_as_list now returns the emitted term so parse_object_list can admit an RDF 1.2 annotation tail on it. Without that, ( # c\n ) {| :q :z |} was still refused by reject_annotation_on_collection while () {| :q :z |} reifies fine — the same divergence, one layer up. With both, the comment spelling is fully equivalent to the NIL-token spelling: same triple, same annotation admission, and the options.rs:38-41 claim is now true for every spelling.

Tests: default_empty_collection_with_comment_is_rdf_nil and default_mode_admits_annotations_on_comment_spelled_empty_collection, both written first and both failed at the pre-fix HEAD (the first with left: [], the second with the annotations on collection objects … are deferred error) — so they're proven non-vacuous. 139/139 lib green with them in.

[bplatz]

docs/audit/burn-down/ROADMAP.md:388 still carries the opposite claim — ingest "drops () objects entirely" and "basic#list-1..4 do NOT green via PR-1". That's where D-13 is tracked, so worth the same one-line correction as this comment.

[bplatz]

Also not sure if you wanted docs/audit/burn-down/ROADMAP.md checked in, along with some of the other docs/audit/* files some of which seem stale.

[aaj3f]

Done — the D-13 bullet under "Post-implementation corrections" (the :388 claim) now reads basic#list-2..4, with a parenthetical noting the () drop was real when that was written and that the #1694 fix greened list-1 and took it off the register (3e2a723).

On the check-in question: ROADMAP.md was already tracked before this PR — git log on main shows it landing through the Wave-0/1/2 outcome commits (81fb52145, 0b84e6e24, e2e63a9f6) — so nothing here newly checks it in. This PR only touches the one line where D-13 is tracked, which I think is in scope because otherwise the register comment and that bullet would contradict each other. You're right that some of the rest of docs/audit/* has drifted, though — I'd just rather not fold a staleness sweep into a correctness fix. That wants to be a docs pass of its own, done deliberately rather than opportunistically.

[aaj3f]

On the JSON-LD twin: folded in rather than filed (a212dfe) — it stayed contained, and a deferred twin of a silent-loss bug is exactly the kind of thing that rots in the backlog.

I verified both of your claims by execution first, and both held: an e2e insert of {"@id": "ex:s2", "ex:items": {"@list": []}} through the public insert surface came back with ex:s2 must exist: [] (zero triples, same shape as the Turtle repro), and the singular position errors exactly as you quoted. Probing the spellings turned up a third site, though: the explicit array spelling [{"@list": []}] doesn't lose the statement — it stores a spurious blank node. expand_node_internal drops any key whose values expand to nothing (expand.rs:644), @list included, so the item expands to {} and downstream treats it as an anonymous node object. Wrong data rather than missing data, which is arguably worse.

So the fold is three small fixes, one per layer: parse_list_values_with_ctx yields the one rdf:nil template (and the singular fallback returns the same term instead of erroring), adapter.rs process_list emits the nil triple on the import/inline-shapes path, and the expansion's drop-empty-keys rule now exempts @list. Each of the three was individually reverted to watch its pin fail before restoring. The parity e2e (it_jsonld_empty_list.rs) asserts the same statements through Turtle () and JSON-LD {"@list": []} store byte-identical graphs. Nothing depended on the zero-triples behavior — the only references anywhere were the two tests pinning it, both updated. -p fluree-db-transact, -p fluree-graph-json-ld, and the api grp_transact/grp_query/grp_query_sparql/grp_import/grp_misc/grp_policy/grp_graphsource groups are all green, plus the W3C suite 36/36.

And you were right about the body — "every ingest surface goes through the one fixed parser path" was just false (JSON-LD ingest never touches the Turtle parser). It's rewritten to name the two pipelines explicitly, with your term.rs:642 note on SPARQL folded in too.

On the << :s :p () >> nit — agreed, leaving it. rtObject excluding collections is the grammar's call, and I'd rather stay conformant than extend the "() is just an IRI" framing into a position RDF 1.2 deliberately excludes.

Thanks for the careful pass on this one — the comment-spelling catch especially; that was a genuine hole.