document-schema.js: section page-furniture slots; wpd-codec lifts its furniture and note bodies

packages/document-schema.js/README.md

@@ -108,7 +108,7 @@ const laidOut = DocumentTreeSchema.parse({
 
 - **Groups** are `{ node, children }` where `node` embeds either an anchor paragraph (heading groups and list-item groups carry the full `ContentParagraph` — runs, formatting, frames — never a projected text label) or a container descriptor: `{ kind: 'section', pageSize, margins }`, `{ kind: 'slide', size, notes }`, `{ kind: 'sheet', name, cells, columns, rows, printSettings }`, `{ kind: 'drawPage', size }`, each tagged with a `kind` the flat container type does not carry, or a shape group's untagged frame descriptor, or — since 4.1.0 — a **construct descriptor** (see [Fidelity constructs](#fidelity-constructs)).
 - **Bare leaves** carry their own `kind` and never `children`. Discrimination is structural on `node`+`children`, not on the presence of a `kind`. Every `ContentBlock` kind is a legal leaf except the two construct boundary markers, which are the flat form's encoding of something the tree carries as a group (see [Constructs in the flat form](#constructs-in-the-flat-form)).
-- **Section groups are mandatory** — one per `ContentSection` — because a section carries pre-layout page geometry (`pageSize`/`margins`, plus the optional `breakType` naming how the section begins — nextPage/continuous/evenPage/oddPage, absent meaning the producer's own default) that a rendered `pages` array cannot hold.
+- **Section groups are mandatory** — one per `ContentSection` — because a section carries pre-layout page geometry (`pageSize`/`margins`, plus the optional `breakType` naming how the section begins — nextPage/continuous/evenPage/oddPage, absent meaning the producer's own default) and its optional page furniture (`headers`/`footers`, per-slot block flows in WordprocessingML's own default/even/first reference vocabulary — ExaDev/documents.js#1128) that a rendered `pages` array cannot hold.
 - **Grouping never crosses container boundaries**: a shape is its own group with its inner blocks grouped inside it (never a slide's paragraphs flattened across its shapes — that is a TOC projection, not a decomposition); a sheet's grid rides on the sheet node with images and embedded documents as children; embedded documents stay intact as one leaf.
 - **Style refs ride on group wrappers only** — a group may carry `style: string` naming a `styles` table entry; `ContentDocument` nodes carry no ref field, so the flat codec-exchange form is always fully materialised.
 

packages/document-schema.js/src/bijection.test.ts

@@ -437,6 +437,28 @@ function corpus(): readonly CorpusEntry[] {
         sections: [{ ...SECTION_GEOMETRY, blocks: [paragraph("body")] }],
       },
     },
+    {
+      name: "wordprocessing section carrying page furniture in every slot (ExaDev/documents.js#1128)",
+      content: {
+        kind: "wordprocessing",
+        metadata: {},
+        sections: [
+          {
+            ...SECTION_GEOMETRY,
+            headers: {
+              default: [paragraph("header default")],
+              even: [paragraph("header even")],
+              first: [paragraph("header first")],
+            },
+            footers: {
+              default: [paragraph("footer default")],
+              even: [paragraph("footer even")],
+            },
+            blocks: [paragraph("body")],
+          },
+        ],
+      },
+    },
     {
       name: "presentation with several shapes, list nesting inside each, and a heading-styled leaf",
       content: {

packages/document-schema.js/src/content-json-schema-defs.test.ts

@@ -54,6 +54,7 @@ import {
   ContentSubpathSchema,
   ContentVectorSchema,
   ContentCellValueSchema,
+  ContentPageFurnitureSchema,
 } from "./content";
 import { CONTENT_DEFS } from "./content-json-schema-defs";
 import {
@@ -123,6 +124,7 @@ import { SourceResidueSchema } from "./source";
 // Comparison strategy: a bare `z.toJSONSchema(SomeSchema)` call, run in isolation, would INLINE every nested schema it encounters (ColorSchema inside ContentRunSchema, AlignmentSchema inside ContentParagraphSchema, etc.) rather than emit the `{ $ref: '#/$defs/X' }` pointers CONTENT_DEFS itself uses -- because those nested schemas aren't registered anywhere. To reproduce the exact cross-reference shape CONTENT_DEFS hand-authors, this test registers the identical set of real schemas under the identical id strings CONTENT_DEFS uses as its own $defs keys, with a `uri` callback matching the `#/$defs/<id>` convention CONTENT_DEFS was written against -- confirmed empirically (see this file's own construction) to make Zod's registry-based multi-schema generation emit exactly that $ref shape for every registered schema referenced from within another. Each per-schema result still carries its own top-level `$schema`/`$id` (since z.toJSONSchema(registry, ...) treats every registered schema as its own standalone root), which CONTENT_DEFS's own nested fragments never have -- those two keys are stripped before comparison, since they're an artefact of testing each fragment as a registry root rather than a real structural difference.
 
 const REGISTERED_SCHEMAS = {
+  ContentPageFurniture: ContentPageFurnitureSchema,
   Color: ColorSchema,
   Box: BoxSchema,
   LayoutFrame: LayoutFrameSchema,

packages/document-schema.js/src/content-json-schema-defs.ts

@@ -603,12 +603,24 @@ export const CONTENT_DEFS: Record<string, JsonSchema> = {
         type: "string",
         enum: ["nextPage", "continuous", "evenPage", "oddPage"],
       },
+      headers: { $ref: "#/$defs/ContentPageFurniture" },
+      footers: { $ref: "#/$defs/ContentPageFurniture" },
       source: { $ref: "#/$defs/SourceResidue" },
       kind: { type: "string", const: "section" },
     },
     required: ["pageSize", "margins", "kind"],
     additionalProperties: false,
   },
+  // The per-slot page-furniture block flows a ContentSection's headers/footers fields carry (src/content.ts's ContentPageFurnitureSchema). Hand-authored here for the same recursive reason as every other block-array shape: the slots hold ContentBlock, which is the hand-written structural guard in Zod and needs its JSON spelling stated alongside.
+  ContentPageFurniture: {
+    type: "object",
+    properties: {
+      default: { type: "array", items: { $ref: "#/$defs/ContentBlock" } },
+      even: { type: "array", items: { $ref: "#/$defs/ContentBlock" } },
+      first: { type: "array", items: { $ref: "#/$defs/ContentBlock" } },
+    },
+    additionalProperties: false,
+  },
   SlideDescriptor: {
     type: "object",
     properties: {

packages/document-schema.js/src/content.ts

@@ -661,6 +661,14 @@ export const ContentBlockSchema: z.ZodType<ContentBlock, ContentBlock> =
     ContentConstructEndSchema,
   ]);
 
+// One furniture kind's per-slot block flows -- see ContentSectionSchema's own headers/footers comment for the slot vocabulary's format evidence. A slot is absent when the section states no furniture for it: an absent default slot with a present even slot is the even/odd-headers shape, not a gap (ExaDev/documents.js#1128).
+export const ContentPageFurnitureSchema = z.object({
+  default: z.lazy(() => z.array(ContentBlockSchema)).optional(),
+  even: z.lazy(() => z.array(ContentBlockSchema)).optional(),
+  first: z.lazy(() => z.array(ContentBlockSchema)).optional(),
+});
+export type ContentPageFurniture = z.infer<typeof ContentPageFurnitureSchema>;
+
 // A docx section: a run of pages sharing one page size/margins (a w:sectPr boundary starts a new one).
 export const ContentSectionSchema = z.object({
   pageSize: PageSizeSchema,
@@ -670,6 +678,9 @@ export const ContentSectionSchema = z.object({
   breakType: z
     .enum(["nextPage", "continuous", "evenPage", "oddPage"])
     .optional(),
+  // The page furniture this section repeats on its rendered pages -- the block flow a header or footer paints -- in the three-slot vocabulary WordprocessingML itself defines (w:headerReference/w:footerReference's own @w:type values default/even/first, with evenAndOddHeaders gating the even slot). Every page-furniture-carrying format narrows onto it: an ODF master page's style:header/style:header-left pair is default/even, its style:header-first the first slot; a WordPerfect D6 header's own occurrence bits (occurs on odd pages / occurs on even pages) state odd-only -> default, even-only -> even, both -> default. Slot-less decorations the vocabulary cannot state -- a watermark is neither header nor footer and owns no parity -- stay outside it. Section-scoped rather than document-level for the reason breakType is: page furniture belongs to the section that renders it, and a document with two sections may give each its own header.
+  headers: ContentPageFurnitureSchema.optional(),
+  footers: ContentPageFurnitureSchema.optional(),
   source: SourceResidueSchema.optional(), // quarantined residue -- opaque text this format carries and no other format interprets (src/source.ts); rides the tree's section descriptor automatically (omit+extend, src/package-node.ts)
 });
 export type ContentSection = z.infer<typeof ContentSectionSchema>;

packages/wpd-codec/README.md

@@ -174,8 +174,13 @@ This is a bet against libwpd's _code_, not against citing its _data_: the charac
 
 Everything below is recognised by the tokeniser and skipped by the fold, so a document containing it still reads — losing that construct's own structure, never the surrounding text. Each is reported through the diagnostic sink rather than passed over in silence.
 
-- **Headers, footers, footnotes, and endnotes** (the 0xD6 and 0xD7 groups). Reported through `wpd/header-footer-dropped` and `wpd/note-dropped`. The text is genuinely recoverable — each function names a General WP Text packet (type 0x08) holding its own function-code stream, which this package's tokeniser and fold would read (the same packet type boxes now resolve their own text content through) — but the flat `ContentDocument` has no page-furniture position for a header or footer and no note position for a footnote body. That body's real home is `document-schema.js`'s tree-only `definitions` table, which a codec producing the flat form cannot reach; it is the same gap `rtf-codec` documents for its own equivalent constructs, and it closes at the schema boundary rather than here — the one gap in this list that stays blocked pending a DocumentTree-producing wpd-codec, distinct from every other entry, which is a parsing-effort gap rather than a schema-shape one.
 - **A box whose content is a presentation, video, macro, sound, or external payload**, and a box relying on its own template's inherited geometry rather than an explicit function-level position/size override. Reported through `wpd/box-content-unresolved` and `wpd/box-frame-unresolved` respectively. An image box IS lifted when its content packet carries a whole PNG or JPEG payload: the packet's raw bytes are scanned by signature and structural walk (`src/stream/image.ts` — the chunk chain to IEND for PNG, the marker segments to EOI for JPEG), never by guessing at a container header, and the span lifts as a `ContentImageBlock` sized by the box's own frame, with an absolute-from-page-edge position carried as the image's `floatPosition`. What stays unresolved: a WPG vector graphic (a distinct binary graphics format `ContentImageBlock`'s `png`/`jpeg`/`svg`/`gif` set cannot hold as recovered, and which this package does not decode — a project on the scale of this package's own WordPerfect reader, not a wiring job) and a native OLE object (see the next bullet). A box with no function-level content override at all — relying entirely on its template's own rendering defaults — is reported through `wpd/box-dropped`, unchanged from before.
+- **Watermarks** (the 0xD6 group's two watermark subfunctions). Reported through `wpd/header-footer-dropped`: a watermark is neither a header nor a footer and owns no parity, so the shared page-furniture vocabulary has no slot for one.
+- **A second header or footer claiming a slot a first already filled.** WordPerfect's own A/B two-slot-per-kind mechanism is a shape the shared one-flow-per-slot vocabulary does not carry; the first function to claim a slot is the one lifted, and the collision is reported through `wpd/header-footer-dropped`.
+
+Headers and footers themselves are LIFTED (ExaDev/documents.js#1128): a D6 function's occurrence bits narrow onto the shared furniture vocabulary's slots (`ContentSection.headers`/`footers`, `default`/`even` — odd-only and both-parities are the default slot, even-only the even slot), its body folded from the General WP Text packet its first prefix ID names. Footnotes and endnotes are anchored in the flat form (a footnote/endnote anchor construct around the reference site, `definition` naming `note-1`, `note-2`, ... in document order) and their bodies are carried by `readWpd` as definitions-table entries in the tree form — the flat `readWpdContent` reports each still-borne body through `wpd/note-dropped`, since the flat `ContentDocument` genuinely has no home for one (`rtf-codec` documents the same split for its own equivalent constructs).
+
+- **A box whose content is an image, presentation, video, macro, sound, or external payload**, and a box relying on its own template's inherited geometry rather than an explicit function-level position/size override. Reported through `wpd/box-content-unresolved` and `wpd/box-frame-unresolved` respectively. An image box's own content resolves to a Graphics Filename prefix packet (type 0x40) whose children carry either WPG vector graphics (a distinct binary graphics format `ContentImageBlock`'s `png`/`jpeg`/`svg`/`gif` set cannot hold as recovered, and which this package does not decode — a project on the scale of this package's own WordPerfect reader, not a wiring job) or a native OLE object (see the next bullet). A box with no function-level content override at all — relying entirely on its template's own rendering defaults — is reported through `wpd/box-dropped`, unchanged from before.
 - **Embedded OLE objects**, stored under the compound file's `PerfectOffice_OBJECTS` storage and named by an image box's Graphics Filename packet's own `0x70`/`0x71` (OLE Object Descriptor / OLE Object Data) children. `archive-codec`'s compound-file reader already reaches that storage, which is how `ooxml.js` recovers a ZIP-payload embedded object — but a WordPerfect OLE object's payload is a native OLE server's own stream rather than a nested document package (`ooxml.js`'s own equivalent case, a classic OLE1 `.bin` payload with no `Package` stream, stays opaque by the identical scope boundary), so recovering one generically is a project in its own right, not a wiring job.
 - **The counter groups** (0xD8, 0xD9, 0xDB, 0xDC): setting, numbering-method, increment and decrement carry no text and change no structure this reader models, so only the Display Number group's own paragraph-number pair is read.
 - **Every merge subfunction other than FIELD** (ASSIGN, CALL, IF, FOR, CASE, and the rest of WordPerfect's own merge scripting language) and **cross-references** (0xD5). A cross-reference's displayed text survives as ordinary text; its target binding does not. Reported through `wpd/merge-code-dropped` and `wpd/cross-reference-flattened`. Unlike FIELD, these can legitimately wrap whole paragraphs of body text as control flow, which the run-scoped field construct's own one-paragraph extent cannot express regardless — a schema gap for a scripting language's control flow, not a parsing gap.

packages/wpd-codec/src/diagnostics.ts

@@ -28,9 +28,11 @@ export const WpdDiagnosticCodes = {
   OutlineNumberRegenerated: "wpd/outline-number-regenerated",
   // The document contains a box: a figure, text box, equation, or graphic. Its contents are not read; see the README's Remaining scope.
   BoxDropped: "wpd/box-dropped",
-  // The document contains a footnote or endnote. Its reference site is where this fires; the note's own text lives in a prefix packet the flat content model has nowhere to put.
+  // The document contains a footnote or endnote whose body the flat ContentDocument has no home for. Its reference anchor IS emitted (a footnote/endnote anchor construct around the reference site); the body is lifted into the tree form's definitions table by readWpd, so this fires only on the flat readWpdContent.
   NoteDropped: "wpd/note-dropped",
-  // The document declares a header, footer, or watermark. The flat content model has no page-furniture position for one.
+  // A note's On/Off reference pair straddled a paragraph boundary, which the run-scoped anchor cannot express.
+  NoteSpansParagraphs: "wpd/note-spans-paragraphs",
+  // The document declares a watermark, or a second header/footer function claims a slot a first already filled (WordPerfect's own A/B two-slot-per-kind mechanism, a shape the shared one-flow-per-slot vocabulary does not carry). A plain header or footer with a resolvable body is NOT dropped -- it lands in ContentSection.headers/footers.
   HeaderFooterDropped: "wpd/header-footer-dropped",
   // The document contains a cross-reference. Its displayed text survives as ordinary text; the reference's own target binding does not.
   CrossReferenceFlattened: "wpd/cross-reference-flattened",

packages/wpd-codec/src/read-structure.test.ts

@@ -608,26 +608,29 @@ describe("document metadata", () => {
 });
 
 describe("constructs this reader does not lift", () => {
-  // Each of these is recognised by the tokeniser and skipped by the fold, so a document containing it still reads -- and says what it lost rather than passing over it in silence.
+  // Each of these is recognised by the tokeniser and skipped by the fold, so a document containing it still reads -- and says what it lost rather than passing over it in silence. Group 0xD6 no longer appears here: a header or footer function is LIFTED into ContentSection.headers/footers (see the page-furniture describe below), and the watermark subfunction -- the one D6 shape the vocabulary has no slot for -- needs its own subgroup, which the empty-occurrence default of these bare fixtures cannot state.
   it.each([
-    [0xdf, WpdDiagnosticCodes.BoxDropped],
-    [0xd7, WpdDiagnosticCodes.NoteDropped],
-    [0xd6, WpdDiagnosticCodes.HeaderFooterDropped],
-    [0xd5, WpdDiagnosticCodes.CrossReferenceFlattened],
-    [0xde, WpdDiagnosticCodes.MergeCodeDropped],
-  ])("reports group %i through the diagnostic sink", (group, code) => {
-    const { document, diagnostics } = readWithDiagnostics([
-      ...text("before"),
-      ...variableFunction({ group, subgroup: 0x00 }),
-      ...text("after"),
-    ]);
-    expect(
-      paragraphsOf(document)[0]
-        ?.runs.map((run) => run.text)
-        .join(""),
-    ).toBe("beforeafter");
-    expect(
-      diagnostics.filter((diagnostic) => diagnostic.code === code),
-    ).toHaveLength(1);
-  });
+    [0xdf, WpdDiagnosticCodes.BoxDropped, 0x00],
+    [0xd7, WpdDiagnosticCodes.NoteDropped, 0x00],
+    [0xd6, WpdDiagnosticCodes.HeaderFooterDropped, 0x04],
+    [0xd5, WpdDiagnosticCodes.CrossReferenceFlattened, 0x00],
+    [0xde, WpdDiagnosticCodes.MergeCodeDropped, 0x00],
+  ])(
+    "reports group %i through the diagnostic sink",
+    (group, code, subgroup) => {
+      const { document, diagnostics } = readWithDiagnostics([
+        ...text("before"),
+        ...variableFunction({ group, subgroup }),
+        ...text("after"),
+      ]);
+      expect(
+        paragraphsOf(document)[0]
+          ?.runs.map((run) => run.text)
+          .join(""),
+      ).toBe("beforeafter");
+      expect(
+        diagnostics.filter((diagnostic) => diagnostic.code === code),
+      ).toHaveLength(1);
+    },
+  );
 });