Content model: where a node came from, and how its words were obtained

[fcbwilliams]

What we need

We're adopting documents.js as the parsing layer for a revenue-diagnostic product that classifies consulting documents (docx/pptx/xlsx/pdf) against a benchmark schema and cites them as evidence in client-facing reports. Two things follow from "cites them as evidence" that the content model has no room for yet.

1. Where a node's content came from (origin)

ContentTable is currently the same type whether it was authored as a native table, read out of a chart's c:numCache, or produced by a layout pass over a sheet. A reader inspecting a document cannot tell which, and neither can anything downstream.

That matters most for the case a consultant hits constantly: the same visual object arrives by two completely different paths. A chart on a slide is either a native c:chart — exact numbers, from the cache — or a screenshot someone pasted in from Excel. To the person flipping through the deck they look identical.

Proposal: an optional origin on content nodes naming what the content is — chart, diagram, table, image, notes, body — set by whichever reader knows, which is the only layer that can. styleId survives round-trip but is format-specific and says nothing for a chart or a diagram.

2. How a node's words were obtained (fidelity)

origin alone is not enough, because those two charts differ in something origin cannot express: one's numbers are verbatim, the other's would be a model's reading of pixels. A pasted chart is origin: 'chart' and derived. One enum cannot carry both facts, so this is a second, independent axis:

• authored — from the file's own markup (c:numCache, a:t, wp:docPr/@descr, a caption paragraph). Verbatim, quotable.
• ocr — character recognition over pixels. Deterministic and reproducible; misreads glyphs but never invents content.
• vision — a vision model's reading. Handles layout, rotation and handwriting far better, and can produce plausible text that is not there.

The distinction is not pedantry — it decides whether a sentence may be quoted in a client deliverable or only referenced. We enforce that downstream, but only if the model tells us which we have.

The three-way split (rather than authored/derived) matters for two concrete reasons: OCR is deterministic, so re-parsing a document yields the same text and therefore the same content hash, where a vision re-run does not; and for dense scanned text OCR is both cheaper and safer, since inventing a contract clause is far worse than misreading one.

3. The annotation itself, and why it cannot live on ContentImageBlock

Our first design put transcript and description on the image block. That is wrong, and a PDF is what proves it.

A chart in a PDF is frequently not an image at all — pdf-codec returns it as ContentVector rects, lines and paths plus positioned text runs. There is no ContentImageBlock to hang anything off, and the thing that needs interpreting is forty sibling nodes, not one. The same is true of a slide where several grouped shapes together form a diagram, and of a SmartArt diagram where the node text is read but the relationships between nodes are not (a five-stage process arrives as five labels with no indication it is a sequence).

So the annotation needs to be:

• node-agnostic — attachable to any content node, not just an image;
• attachable to a container, because the subject is often a set of sibling nodes rather than one.

Shape we'd propose, as an optional field group in the same spirit as the existing sourcePath / frames / source residue that every leaf variant already carries:

interpretation?: {
  transcript?: { text: string; confidence: 'high' | 'medium' | 'low' };
  description?: string;
  fidelity: 'ocr' | 'vision';
  by?: { model: string; at: string };
}

Two fields rather than one marked string, because they carry different rights: a transcription is the document's own words that happen to be in pixels — quotable — while a description is the model's words about the document, referenceable and never quotable. Both can be present for one subject (a slide screenshot with a title above a chart). A flag would let a consumer forget the difference; two fields cannot be confused.

confidence is a three-value enum deliberately, not a 0–1 float: a self-reported numeric confidence from a VLM does not calibrate, and a float invites downstream thresholds that look principled and are not.

4. caption

ContentImageBlock already carries altText (thank you — that saved us a request). What it has no room for is the visible caption beside a figure, which is usually far more informative than alt text and, unlike alt text, is usually present.

We've implemented the docx side of this (a Caption-styled paragraph, preferring the one below the figure, claimed by at most one figure) and will offer it as a PR separately. The schema question is where the association lands: our own answer is that a caption is associated with the figure rather than copied into it, because it is already a paragraph in its own right and copying it would duplicate it in every flat-text projection and search index built from the document.

What we are NOT asking for

Keep the codecs model-free and offline. No OCR, no vision calls, no network. That property is worth protecting, and pdf-codec's README is right to boast about it. We'll call the models; we just need somewhere to put what comes back, and — for the vector and scanned-page cases — a way to get pixels out (that is a separate request about page rasterisation).

Why it can't be done outside the schema

We could hold all of this in a side table keyed by sourcePath. We'd rather not, and the reason is in your own repo history: the two-tree design that correlated a separate LayoutDocument back to ContentDocument nodes by matching sourcePath strings was replaced by putting frames on the nodes themselves. This is the same shape of problem and would earn the same fix.

Happy to write this as a PR rather than leave it as a request if the shape above is roughly right — say which parts you'd want differently first, since it touches every content variant.

[fcbwilliams]

A fifth thing, which belongs here rather than in a PR: an inline image's position within its paragraph

Raising it here because it is a schema question, not a reader one, and read.ts's own comment already says so:

ContentRun has no field to carry an inline image or embedded object (unlike ContentShape's blocks list in pptx) — media found inside a paragraph's own runs is therefore surfaced as its own sibling block … at the cost of losing each one's exact character-level position within that paragraph's text — a real, bounded scope narrowing forced by ContentParagraph's own shape, not a silent drop.

Agreed on all of that, including that it is bounded and honestly documented. What I want to add is where it actually costs something, since we hit it.

Why the position matters to a consumer

The lifted-sibling design is exactly right for a figure: a picture in its own paragraph, caption below, position within the paragraph meaningless. That is most images in most documents.

It is wrong for the other case — a small image inline in a sentence. A status icon mid-row, an inline logo, a signature graphic, a checkbox glyph. For those, the paragraph's text and the image are one unit of meaning, and lifting the image out turns "Status: ✅ approved by [signature]" into a paragraph and two orphaned blocks with nothing tying them together.

Two concrete consequences we ran into:

Chunking. We split a document into units for classification and retrieval. A paragraph and its lifted images arrive as separate blocks, so either the images become their own contentless chunks, or we re-associate them by adjacency — which is guesswork the reader already knew the answer to. Note this is the same guesswork the caption association in #1201 avoids by recording what the reader knew at the time.

Evidence citation. We cite document content in client-facing reports. "The image after paragraph 12" is a citation nobody can check; "the image in paragraph 12, after 'approved by'" is.

What we do, and why it is not a PR

In our own OOXML reader an inline image becomes a child of the text node rather than a sibling. Reading order is unchanged (a pre-order walk visits a node's own content before its children), but the text keeps its position in the block list and the image hangs off it in place.

The reason that is not a PR against this repo is that our tree shape permits it and ContentParagraph does not — it is a flat runs: ContentRun[]. Making it work here needs one of:

• a ContentRun variant that carries an image (changes what a "run" is for every codec and every writer);
• an optional anchorRunIndex/anchorOffset on the lifted ContentImageBlock, pointing back at the position it came from — additive, no existing shape changes, and enough for both consequences above;
• or leaving it exactly as it is, and having consumers who care re-associate by adjacency.

The second is the one I would build, because it is additive and because it is the same shape of answer as caption in #1201: record what the reader knew, next to the thing it knew it about, without moving anything. But this is your model and a run-level change touches every format — so a decision, not a patch.

Happy to implement whichever you pick, or to leave it alone: we can live with adjacency re-association, and it is a much smaller cost to us than the interpretation model above.

While I am here — two smaller observations from the same work

ContentSlide.notes is a plain string. Speaker notes lose all paragraph structure on the way in, where the notes slide's own a:p elements carry it. For a deck where notes are the substance (a script, a set of talking points), that flattens the one part a reader most wants structured. Not asking for anything yet — just flagging that it is the only text-bearing field in the content model that is a bare string.

isContentBlock and the image format enum disagree. ContentImageBlockSchema.format accepts png | jpeg | svg | gif, but the runtime type guard at content.ts only admits png/jpeg — so a schema-valid SVG or GIF image block fails isContentBlock. Looks unintentional given the enum's own comment explains why svg/gif were added. Say the word and I will send it as a one-line PR.

[Mearman]

fidelity: 'ocr' | 'vision' at the top of interpretation breaks once transcript and description are both populated with different provenance, e.g. deterministic OCR on the axis labels, vision-model description of the chart. description is always model-generated prose, there's no deterministic path to one, so fidelity on it is redundant. Move it onto transcript only: transcript: { text, confidence, mechanism: 'deterministic' | 'model' }, drop fidelity from description.

Also rename the enum. 'ocr' | 'vision' conflates which product produced the text with whether it's quotable, a vision model asked to transcribe verbatim is still OCR functionally, it just isn't reproducible. mechanism: 'deterministic' | 'model' names what actually matters.

Keep this PR to representation only. Your own "not asking for" section already rules out documents.js calling a model, don't let the implementation grow a provider hook for actually running OCR alongside it. Nobody's asked for that.

Question, not a change: given the classification-against-a-benchmark use case, is there a third kind needed alongside transcript/description, a label that isn't a reading of anything ("this is a signature block")? Or handled outside interpretation entirely?

[fcbwilliams]

Decisions from our side, so this is concrete rather than open-ended

Answering my own questions above, since they were ours to answer:

Two axes, confirmed. origin and fidelity stay independent. A pasted chart screenshot is origin: 'chart' and derived; one enum cannot express that, and it is the case that made us look at this in the first place.

interpretation attaches where a container's content is the thing being described — not to arbitrary containers. So: a slide ContentShape whose blocks are the diagram, yes. A ContentTable read from a chart, yes. A ContentSection that merely contains a chart, no.

For PDF that rule has no candidate, which is the interesting part: a vector chart is dozens of sibling ContentVector items with nothing whose extent equals the chart. Rather than introduce a new grouping node into the content tree, we would attach it to a document-outline.js region — segmentPdfRegions already classifies page items into column/table/figure and already attaches captions to figures geometrically, so a region is exactly "a container whose content represents the whole thing", and it lives outside the content tree so nothing structural changes.

Inline image position: the additive option. Optional anchorRunIndex (and anchorOffset) on the lifted ContentImageBlock, pointing back at where it came from. No existing shape changes, consumers that do not care ignore it, and the position becomes recoverable rather than structural. Same shape of answer as caption in #1201 — record what the reader knew, beside the thing it knew it about. We are explicitly not asking for an image-carrying ContentRun: that changes what a run is for every codec and writer, and the cost is not worth the fidelity for our use.

ContentSlide.notes: leave it. We flagged it as a bare string, but we do not read notes structurally ourselves yet, and asking for something we cannot use would be a poor trade. Withdrawn until we can.

isContentBlock svg/gif: we will send that as a one-line PR — verified in source, the guard admits png/jpeg while ContentImageBlockSchema.format accepts four.

Happy to implement the interpretation model itself against this shape. Say if any of it should be different before we start — particularly the region-rather-than-node choice for PDF, which is the one that touches a second package.

[fcbwilliams]

Correction on the isContentBlock svg/gif point — it needs a decision, not a patch

I said above I would send that as a one-line PR. I started it and then backed out, because I had mischaracterised it.

There is an existing test that asserts the current behaviour deliberately — content.test.ts, in the case named "rejects a malformed block at every level":

expect(
  isContentBlock({ kind: "image", format: "gif", base64: "AA==", widthPt: 1, heightPt: 1 }),
).toBe(false);

So the guard admitting only png/jpeg is not obviously drift — someone wrote that expectation on purpose. Which leaves two readings, and only you can say which is right:

1. The guard is correct and the schema is loose. isContentBlock is the structural guard used via z.custom for table cells and embedded-object documents; perhaps only genuinely decodable raster formats belong in those positions, and a gif block there really is malformed.
2. The guard is stale. ContentImageBlockSchema.format's own comment says svg/gif were added for epub's manifest kinds and that "a codec whose reader cannot yet decode one of the four degrades to alt text with a diagnostic" — which reads like the model is meant to admit all four everywhere, with degradation handled at the codec.

Reading 2 is what I assumed, but the test is evidence for 1 and I should not flip a deliberate assertion on an assumption.

If it is 2, the fix worth having is not adding two strings to the guard — it is removing the second list entirely, so the guard and the schema read one shared IMAGE_FORMATS tuple and cannot diverge again. That is what I had built when I hit the test. Happy to finish it on a word from you; leaving it alone otherwise.

No impact on us either way: we only produce png/jpeg.