Add an offline index-version graph diagnostic - #1622
Conversation
Decodes every .fir6 root in a copied roots directory, rebuilds the DAG they form through prev_index, and reports which of three shapes a ledger is in: one broken chain, multiple lineages, or roots never linked in at all. Those have different fixes upstream, which is why the distinction is worth a tool rather than an inspection. Carried over from #1600, whose fix is superseded by the reclamation work now on main. The diagnostic is not — nothing on main reconstructs the chain offline, and it is what ended the investigation. On a live ledger it reported 226 roots, 15 reachable, 211 orphaned, with fork points 0 and 211 distinct index_t values, which eliminated both the publish-race and built-twice explanations; two roots carried no prev_index at all, at 6739 and 8627, neither of them genesis. Six earlier theories had each been eliminated by measurement before this named the cause directly. Run against a roots directory copied out of a pod: cargo run -p fluree-db-indexer --example root_graph -- <roots-dir> [head-digest] Diagnostic only — an example target, not built by a plain cargo build, and not shipped behaviour. Refs #1600, #1548
| .iter() | ||
| .filter_map(|(d, (t, p))| { | ||
| p.as_ref() | ||
| .filter(|p| !nodes.contains_key(*p)) |
There was a problem hiding this comment.
Undecodable roots never enter nodes, so a corrupt root is reported as MISSING here — truncating vs deleting the same mid-chain root gives byte-identical output. That aims shape 1 at GC delete-ordering when the cause is a bad blob; worth tracking undecodable digests separately and gating the shape-1 verdict on undecodable == 0.
| let mut seen = HashSet::new(); | ||
| let mut cur = Some(head.clone()); | ||
| while let Some(d) = cur { | ||
| let Some((_, prev)) = nodes.get(&d) else { |
There was a problem hiding this comment.
A head digest that isn't on disk breaks on the first iteration and prints 0 of 202 roots (202 UNREACHABLE) with no warning, so a typo or a CID pasted instead of raw hex is indistinguishable from total corruption. Worth failing loudly when the supplied head isn't in nodes.
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
Since I first drafted this review, @bplatz's review comments landed, so I double-checked everything below.:
Brian's :150 note and my head-digest item below are the same defect approached from opposite ends — his case is the mistyped digest, mine is that the input the header documents can never match at all — so please read mine as a +1 with an addendum rather than a second, separate ask. His :120 point about undecodable roots being reported as MISSING is one I hadn't caught and I think he's right, particularly the suggestion to gate the shape-1 verdict on undecodable == 0; it composes with my note at :42 about read failures, since both are cases of the tool folding a distinct cause into its scariest verdict.
To say a bit more about head-digest handling at :137, which is also Brian's :150 ask. The header tells the operator to pass the nameservice's index_head_id, but that value stringifies as a base32 CIDv1 (fluree-db-core/src/content_id.rs:156-161) while nodes is keyed by the hex digest file stem (fluree-db-core/src/storage.rs:1114), so the documented input misses on the first lookup, the walk ends with seen empty, and the tool prints total orphaning.
I ran it to be sure: same healthy one-root directory reports 1 of 1 roots (0 UNREACHABLE) with no head argument and 0 of 1 roots (1 UNREACHABLE) with a CID-form head, silently. A three-line "that head isn't a root in this directory" guard closes it.
The second note I'd care about is the fork-signature block at :169-199 — it's computed over unreachable roots only, so an ordinary fork (winner reachable, loser orphaned) contributes exactly one root at that t and gets filtered out by n > 1; I don't think that changed the #1600 conclusion, since fork points: 0 is what actually carried it, but a line labelled "fork signature" that can't fire on a fork will get trusted next time. Everything else in the list is small.
One cross-PR note, no action needed here: this tool's end-of-chain rule ("digest not on disk → chain ends") deliberately differs from main's walk, which errors when the head can't be read and only ends gracefully on a missing parent after store.has() confirms it. Your #1643 is restructuring that walk into PrevIndexChainWalk and has an open question about end-of-chain semantics at the head position, so it's probably worth making the two tell the same story once that lands.
Adherence to repo commitments:
- Patterns/abstractions: ✔ Follows the repo's existing diagnostic-example convention across four crates; doesn't reinvent the on-main chain walk (which it couldn't call anyway —
pub(crate)), and doesn't duplicate anything the CLI can do. Only altitude question is whether it belongs influree-db-binary-index/examples/instead, since that's its sole dependency. - Performance (speed first, memory second): ✔ No performance-degradation risk. Example target only —
cargo builddoesn't build it, nothing links it, no engine or per-flake path is touched, no dependency added. The sole cost isclippy --all-targetscompiling it in CI. - Testing:
⚠️ No tests, which matches how the repo treats its other diagnostic examples (concurrent_add_repro.rssays so in its own header), so I'm not asking for any. Worth knowing that the classification logic is therefore unguarded — if we ever want it pinned, it has to move into a lib module first. - Conventions: ✔
fmtandclippy -D warningsboth clean locally on the example target
Verified locally at branch HEAD: cargo clippy -p fluree-db-indexer --example root_graph -- -D warnings → exit 0 with the example target confirmed in the output; cargo fmt -p fluree-db-indexer -- --check → exit 0; built the example and ran it against a real index/roots/ directory (healthy case correct), against the same directory with a CID-form head (false 100%-orphaned reading), and against a synthetic four-root directory with tied index_t five times (head selection varied across runs).
| } | ||
|
|
||
| // Reachability from the head: what GC can actually see. | ||
| let head = head_arg.or_else(|| { |
There was a problem hiding this comment.
Worth fixing before merge
fluree-db-indexer/examples/root_graph.rs:137 — blocking-ish (small fix). The [head-digest] argument as the header documents it can't match, and when it doesn't the tool reports the most alarming possible answer with no warning.
The header at :23-25 tells the operator to pass "the published index head (the nameservice's index_head_id)". That value is a ContentId (fluree-db-consensus/src/raft/state_machine.rs:1050), and everywhere it gets stringified it goes through Display — e.g. fluree-db-cli/src/remote_client.rs:2579 does cid.to_string() — which renders base32-lower CIDv1 (fluree-db-core/src/content_id.rs:156-161). But nodes is keyed by file stem, i.e. digest_hex() (fluree-db-core/src/storage.rs:1114). Those two strings never coincide, so nodes.get(&d) at :150 misses on the very first iteration, the walk breaks with seen empty, and :159-165 prints total orphaning.
I ran this against a real single-root index/roots/ directory to be sure I wasn't inventing it. With no head argument: reachable from head 23e139c4f5c4503d : 1 of 1 roots (0 UNREACHABLE). Same directory, same binary, head passed in CID form: reachable from head bafkreibd4e44yrh : 0 of 1 roots (1 UNREACHABLE). A perfectly healthy ledger reporting 100% orphaned, silently — which is exactly the reading that starts the next week-long investigation.
The cheap fix is to refuse to guess: if the supplied head isn't in nodes, say so and stop rather than reporting a walk of length zero.
if let Some(h) = &head_arg {
if !nodes.contains_key(h) {
eprintln!("!! head {h} is not a root in this directory — pass the hex digest (the .fir6 file stem), not the CID form of index_head_id");
std::process::exit(2);
}
}Fuller version if we want to be kind to whoever is pasting from fluree info at 2am: try ContentId::from_str(&arg) first and fall back to its digest_hex(), so both forms work.
Addendum, post-drafting: @bplatz's comment at :150 asks for the same loud failure from the typo end. The addition this note carries beyond his: it isn't only a mistyped digest that walks into the silent 0 of N — the value the module header tells the operator to pass (index_head_id) stringifies as base32 CIDv1 and can structurally never match the hex file stems, so the guard wants either a CID→digest_hex() translation on the way in or the header rewritten to say "pass the hex digest file stem." Either closes both his case and mine.
|
|
||
| // How do the unreachable ones distribute over index_t? A contiguous block | ||
| // means one truncation event; a scatter means repeated forking. | ||
| let mut unreachable_ts: Vec<i64> = nodes |
There was a problem hiding this comment.
Worth fixing before merge
fluree-db-indexer/examples/root_graph.rs:169-199 — should fix. The "fork signature" line is computed over unreachable roots only, so it can't actually see the ordinary fork — which matters because #1600's writeup leans on it.
unreachable_ts is filtered by !seen.contains(*d) at :171, and dupes at :187 keeps only t values with n > 1. In the textbook fork — one parent, two children at the same index_t, one published and one abandoned — the winner is reachable and only the loser lands in unreachable_ts. That's a count of 1 at that t, the n > 1 filter drops it, and the line prints nothing. So "distinct unreachable index_t: 211 (so 0 share a t with another root)" does not rule out competing builds; the thing that actually ruled it out in #1600 was the separate fork points: 0 line at :99, which is sound.
I don't think this changed the #1600 conclusion — fork points: 0 carried it — but a line labelled "fork signature" that can't fire on a fork is the kind of thing that gets trusted next time. Building the multiplicity map over all of nodes rather than the unreachable subset makes the label true, and it's a one-line change to what gets folded.
Two smaller things in the same block, since they're right there: the count in :189-191 reads unreachable_ts.len() - distinct.len(), which is the number of excess roots rather than the number of roots sharing a t (with three roots at one t it says "2 share a t with another root", where the honest answer is 3); and :88, :100, :128, :187 all take(n) without printing an "… and N more", so a truncated preview looks the same as a complete list.
| "roots with no prev_index (expect 1 = genesis): {}", | ||
| genesis.len() | ||
| ); | ||
| for (t, d) in genesis.iter().take(5) { |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:88 — optional. Output isn't reproducible run to run, which is a shame for a tool whose main use is comparing two measurements. genesis, forks, and dangling are all built from HashMap::iter(), so which five or eight get previewed shuffles between runs; and :137-142 picks the head with max_by_key over the same iterator, so when two roots tie on index_t — precisely the fork case the tool exists to detect — the head is chosen arbitrarily. I built a directory of four roots sharing an index_t and ran it five times: the head came out dddd…, dddd…, bbbb…, cccc…, cccc…, and diffing two full runs shows the genesis listing reordered as well. Sorting each preview by (index_t, digest) and breaking the head tie on digest would fix both, and it might be worth an explicit note in the output when the max-t is tied, since that tie is itself a finding.
| let bytes = std::fs::read(&path).expect("read root"); | ||
| match IndexRoot::decode(&bytes) { | ||
| Ok(root) => { | ||
| let prev = root.prev_index.as_ref().map(|p| p.id.digest_hex()); |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:55 — optional, and probably the highest value-per-line suggestion here. BinaryPrevIndexRef carries t: i64 next to id (fluree-db-binary-index/src/format/wire_helpers.rs:87-92), and we throw it away. Two things fall out for free if we keep it: a dangling pointer at :128-134 could print the index_t of the missing parent rather than just an opaque digest, which is exactly the "where is the hole" answer shape 1 is asking for; and comparing prev.t against the parent's own decoded index_t when the parent is present is a cheap consistency check on the chain metadata itself — a disagreement there is a fourth shape the tool currently can't see.
| let mut nodes: HashMap<String, (i64, Option<String>)> = HashMap::new(); | ||
| let mut undecodable = 0usize; | ||
|
|
||
| for entry in std::fs::read_dir(&dir).expect("read roots dir") { |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:42-52 — optional. The error handling is asymmetric in the direction I'd expect to bite: a root that fails to decode is counted and survived (:58-61), but a root that fails to read takes the whole run down via expect("read root") at :52 (same for read_dir at :42 and the entry at :43). Given the tool is pointed at a directory copied out of a pod — possibly mid-GC, possibly a partial rsync — one unreadable file killing a 226-root survey seems like the wrong trade. Folding read failures into the same counter (renamed to something like unreadable) would keep the run alive and make the count itself diagnostic.
| .map(|(d, (t, _))| (*t, d.clone())) | ||
| .collect(); | ||
| println!( | ||
| "roots with no prev_index (expect 1 = genesis): {}", |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:85 — nit. "expect 1 = genesis" isn't right for the healthy steady state. Once GC has legitimately truncated past genesis the correct answer is 0, which is why #1600's writeup had to spend a paragraph explaining that the 0 in the "after" output wasn't a regression. Something like "expect 0 or 1 — 0 once GC has truncated past genesis" would carry that in the output instead of in prose.
| println!("undecodable : {undecodable}"); | ||
|
|
||
| // Which roots are pointed AT by some other root? | ||
| let mut referenced: HashSet<&String> = HashSet::new(); |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:69 — nit. referenced is inserted into at :72 and never read. Looks like a leftover from an earlier shape of the analysis.
| //! known; supplying it lets the tool report reachability from the real head rather | ||
| //! than guessing the newest by `index_t`. | ||
|
|
||
| use fluree_db_binary_index::IndexRoot; |
There was a problem hiding this comment.
Optional / nits
fluree-db-indexer/examples/root_graph.rs:27 — more of a question than a suggestion. The file's only non-std import is fluree_db_binary_index::IndexRoot — nothing from fluree-db-indexer is used at all. Would fluree-db-binary-index/examples/ be a better home? The argument for where it is now is thematic (this is a GC/chain question and gc/ lives here), and that's a real argument. The argument against is that during an incident cargo run -p fluree-db-indexer --example root_graph builds core, novelty, spatial, nameservice and the indexer to decode some file headers, possibly on the box that's already unhappy. Entirely reasonable to leave it if you'd rather keep it next to the code it reasons about.
| @@ -0,0 +1,203 @@ | |||
| //! Reconstruct a ledger's real index-version graph from its root blobs. | |||
There was a problem hiding this comment.
Praise (worth preserving)
fluree-db-indexer/examples/root_graph.rs:1-25 — the module header is the best part of this PR and the strongest reason to keep the file in the repo. Naming the three shapes and what each implies upstream ("GC deletes oldest-first precisely to make this impossible, so finding it means something deletes out of order") is what makes the output actionable six months from now by someone who wasn't in the incident.
| .map(|(d, _)| d.clone()) | ||
| }); | ||
| if let Some(head) = head { | ||
| // Count only digests that are actually PRESENT on disk. Walking into a |
There was a problem hiding this comment.
Praise (worth preserving)
fluree-db-indexer/examples/root_graph.rs:144-151 — counting only digests actually present on disk, with the comment explaining that walking into a missing parent would make seen exceed nodes and underflow the subtraction, is exactly right, and the cycle guard at :153-156 is the sort of thing that gets skipped in a throwaway tool and then bites.
The module header tells the operator to pass the nameservice's index_head_id, but that value stringifies as a base32 CIDv1 while nodes is keyed by the hex digest that names the .fir6 file. The two strings can never coincide, so the documented input missed on the first lookup, the walk ended with seen empty, and a healthy ledger was reported as 100% orphaned with no warning. A mistyped digest took the same silent path. resolve_head now tries an exact key match first, then ContentId::from_str followed by digest_hex, so both spellings work. Hex is tried first on purpose: multibase reads a leading 'f' as base16, so a hex digest beginning with 'f' could otherwise parse as a CID. A head that resolves to neither prints what to pass instead and exits 2, rather than reporting a zero-length walk as total orphaning. Verified against synthetic root directories built from IndexRoot::encode: no head and hex head both give 4 of 4, the CID form now resolves to 4 of 4 instead of 0 of 4, and a bogus digest exits 2.
A root that fails to decode never enters `nodes`, so every downstream check saw it as absent. A dangling prev pointer into a truncated blob printed `-> MISSING`, byte-identical to one into a deleted blob, which aims shape 1 at GC delete-ordering when the cause is a bad blob and nothing deleted anything. Keep the undecodable digests rather than counting them, and use the set in the four places the distinction matters: dangling lines label MISSING vs UNDECODABLE and the count splits the same way; the shape-1 verdict is withheld while anything on disk fails to decode; the reachability walk says when it stopped at an undecodable root instead of ending as if truncated; and resolve_head reports a head that is on disk but did not decode separately from one that is not there, for both accepted spellings. Verified on two directories differing only in whether the t=2 root was deleted or truncated in place, which previously produced the same output.
The fork-signature block folded its multiplicity map over the unreachable subset, so it could not fire on the fork it was named for: in the ordinary case one parent, two children at one index_t, the winner is published and therefore reachable, only the loser is orphaned, and the n > 1 filter drops a count of 1. A line reading "0 share a t with another root" was printed for a directory that plainly contains a fork. Count over every root and show the reachable/unreachable split per index_t. The unreachable distribution keeps a line of its own, since it answers the separate question of whether the orphans are a truncated prefix. The share-count was also wrong in its own terms: len - distinct is the number of excess roots, so three roots at one t reported "2 share a t". Sum the group sizes instead. Every preview here is a take(n) with no closing line, so a complete list and a cut-off list looked alike. and_n_more() closes all four. Verified on a synthetic fork (silent before, now reports index_t=2: 2 roots, 1 unreachable), four roots tied at one t (reports 4 roots, not 2 or 3), and ten unchained roots (prints five, then "... and 5 more").
genesis, forks and dangling are all collected out of a HashMap, so which roots a take(n) preview happened to show, and the order it showed them in, changed between two runs over an unchanged directory. The tool's main use is diffing two measurements, which that defeats. Sort each listing by (index_t, digest), and the children inside a fork line too, since those were pushed in nodes iteration order. Head selection had the same problem with worse consequences: max_by_key picks arbitrarily among ties, and a tie at the newest index_t is exactly the fork case, so the tool could pick a different side of the fork on each run and report different reachability. Break the tie on digest, and say the tie happened instead of silently guessing. Over 12 runs on four roots tied at one index_t the head came out four different ways before this and one way after. Full output is now identical across 5 runs on all six fixtures.
BinaryPrevIndexRef carries the parent's t next to its CID and we were dropping it on decode. Two things come back for free by keeping it. A dangling prev pointer can name the index_t of the parent that is gone, which is the "where is the hole" answer shape 1 is asking for, and it does not need the blob that is missing to say it. Where the parent IS present, the child's claimed prev_index.t and the parent's own index_t are a consistency check on the chain metadata. A disagreement is a fourth shape: the pointer still resolves, so the walk succeeds, no pointer dangles, nothing forks, and every existing line reads clean while one of the two values is wrong. The header documents it as shape 4, and the check prints its count even at zero so a clean run is not mistaken for a check that never ran. nodes moves from a tuple to a Node struct: the tuple would otherwise have become (i64, Option<(String, i64)>), which no call site could read. That is most of the diff. Verified on a fixture whose t=3 root points at the right t=2 digest but claims prev_index.t = 99: reported by the new line, invisible in every other.
Error handling was asymmetric in the direction that bites: a root that failed to decode was counted and survived, but one that failed to read took the whole run down through expect(). This is pointed at a directory copied out of a pod, possibly mid-GC, possibly a partial rsync, so one unreadable file killing a 226-root survey is the wrong trade. Fold read failures in beside decode failures. undecodable becomes unusable, a digest -> reason map rather than a set, so the two share every downstream check (they both mean "on disk but no header", which is all those checks care about) while the printed cause still says which. A failed directory entry has no name to file it under, so it counts into unlistable, printed only when nonzero. read_dir stays fatal since there is nothing to survey, but exits 2 with a message instead of a panic. Verified with a chmod 000 root mid-chain: panic and no output at all before, now unreadable: 1, the dangling pointer labelled UNREADABLE, shape 1 withheld, and the remaining 2 of 3 roots surveyed.
"expect 1 = genesis" is wrong for the healthy steady state: genesis has no prev_index by construction, but once GC has legitimately truncated past genesis the correct answer is 0, which is why #1600's writeup had to explain in prose that the 0 in its "after" output was not a regression. Say it in the output instead, and note in the comment that two or more is the case that means something.
`referenced` was inserted into and never read, a leftover from an earlier shape of the analysis; `insert` counts as a use, so nothing warned. The comment above it described `referenced` rather than the `prev_targets` map that survives, so it now describes that instead. No output change on any of the eight fixtures.
The file imports IndexRoot and ContentId and nothing from fluree-db-indexer, so it now lives next to the format it decodes. Only the usage line in the header changes with it.
|
@bplatz, @aaj3f all nine actionable items are in, one commit each, so they're reviewable separately:
|
4 participants
Decodes every .fir6 root in a copied roots directory, rebuilds the DAG they form through prev_index, and reports which of three shapes a ledger is in: one broken chain, multiple lineages, or roots never linked in at all. Those have different fixes upstream, which is why the distinction is worth a tool rather than an inspection.
Carried over from #1600, whose fix is superseded by the reclamation work now on main. The diagnostic is not — nothing on main reconstructs the chain offline, and it is what ended the investigation. On a live ledger it reported 226 roots, 15 reachable, 211 orphaned, with fork points 0 and 211 distinct index_t values, which eliminated both the publish-race and built-twice explanations; two roots carried no prev_index at all, at 6739 and 8627, neither of them genesis. Six earlier theories had each been eliminated by measurement before this named the cause directly.
Run against a roots directory copied out of a pod:
Diagnostic only — an example target, not built by a plain cargo build, and not shipped behaviour.
Refs #1600, #1548