corpora: validate doc, rtf, xlsx drawings, and compute against real-producer output

packages/doc-codec/.gitignore

@@ -5,3 +5,4 @@ dist/
 coverage/
 .eslintcache
 .turbo/
+test/corpus/

packages/doc-codec/README.md

@@ -44,7 +44,7 @@ Built and shipped, on the write side — see [Writing](#writing) for the full sc
 | **Table and numbering style formatting**                                                  | `STD.grLPUpxSw` is resolved only for paragraph and character styles (`stk` 1/2, see the row above) -- a table style's `StkTableGRLPUPX` (its own `TAPX` plus `CNFC`-conditional per-region formatting, [MS-DOC] 2.4.6.6's own table-specific steps 6-9) and a numbering style's `StkListGRLPUPX` are neither read nor written. A table applying a named table style, or a paragraph using a numbering style rather than direct list formatting, reports none of the formatting either would supply.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | **Subdocuments, on write; text boxes, on either side**                                    | Footnotes, endnotes, comments, and headers/footers are now read (see the three read-side bullets above); none is written — `writeDocContent` still only ever produces a main document with `ccpFtn`/`ccpHdd`/`ccpAtn`/`ccpEdn` all zero, and a `ContentDocument` carries nowhere on its own `ContentSection.blocks` to state a header/footer's own content distinctly from the body's, which a real inverse would need. Text boxes are not read or written in either direction: `PlcfTxbxTxt` divides its own subdocument into stories the identical way the note-text plexes above do, but a text box's own anchor rides the drawn-object mechanism ([Images](#images)'s own scope note), which this package does not resolve.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | **Extended and user-defined document properties**                                         | `title`/`subject`/`author`/`keywords`/`createdIso`/`modifiedIso` are read from and written to a `"\x05SummaryInformation"` stream when present (see [Metadata](#metadata)); the sibling `"\x05DocumentSummaryInformation"` stream (company, manager, and custom user-defined properties) is not read or written at all.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
-| **Encryption**                                                  | RC4-encrypted ([MS-DOC] 2.2.6.2, ExaDev/documents.js#1113) and XOR-obfuscated (2.2.6.1, #922) documents are both decrypted given a password -- `readDocContent`/`readDocStreams` take an optional `password`, verified against the Table stream's own EncryptionHeader for RC4, or against `FibBase.lKey` directly for XOR obfuscation (see [Encryption](#encryption) below). RC4 CryptoAPI (a different, newer EncryptionHeader shape) still refuses with a `DocUnsupportedError` rather than being read as plaintext or misread as the wrong scheme. `writeDocContent` never encrypts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| **Encryption**                                                                            | RC4-encrypted ([MS-DOC] 2.2.6.2, ExaDev/documents.js#1113) and XOR-obfuscated (2.2.6.1, #922) documents are both decrypted given a password -- `readDocContent`/`readDocStreams` take an optional `password`, verified against the Table stream's own EncryptionHeader for RC4, or against `FibBase.lKey` directly for XOR obfuscation (see [Encryption](#encryption) below). RC4 CryptoAPI (a different, newer EncryptionHeader shape) still refuses with a `DocUnsupportedError` rather than being read as plaintext or misread as the wrong scheme. `writeDocContent` never encrypts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
 | **`sprmPHugePapx` / `sprmPTableProps`**                                                   | Paragraph properties stored indirectly in the Data stream are not followed, so such a paragraph reads with fewer properties than it states. [MS-DOC] 2.4.3's own Overview of Tables text names `sprmPTableProps` as a real, legal alternative to `sprmTDefTable` some applications process — but a real producer's row mark is not shown to prefer it: a genuine LibreOffice-authored `.doc` table's own row mark states its TAP through the identical direct `sprmTDefTable` this package's reader and writer already use (confirmed by parsing a LibreOffice 26.2.5.2-authored table's raw `PapxFkp` bytes; see [ExaDev/documents.js#892](https://github.com/ExaDev/documents.js/issues/892)), matching 2.4.3's own compatibility guidance ("An application SHOULD use sprmTDefTable to define table cells for applications that do not process sprmPTableProps"). `writeDocContent` never writes an indirect Papx.                                                                                                                                                                                                                                                                                                                                                                                                                                              |
 | **Hyperlinks and fields**                                                                 | `ContentRun.hyperlink`, footnote/comment/annotation references, and every other field or anchor character are read as plain text or dropped (see [What is converted](#what-is-converted)) and are not written.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
 | **Every FIB field beyond what this package's own reader needs**                           | `writeDocContent` populates only the fc/lcb pairs its own reader consults (the style sheet, the two property bin tables, the Clx, the font table). Roughly 140 other `FibRgFcLcb97` pairs — `SttbfAssoc`, `Dop`, the printer-driver structures among them — are left zero, which is the format's own "undefined, MUST be ignored" contract for most of them, but not a certification that every third-party [MS-DOC] reader accepts the result; see `fib/write.ts`'s own note.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
@@ -345,7 +345,7 @@ The writer is verified the opposite way: `src/write.test.ts` reads every documen
 
 A `describe("writeDocContent multiple sections")` block (`ExaDev/documents.js#971`) covers two and three sections each round-tripping their own distinct page size and margins independently, and a non-final section whose own last block is a table still closing on a genuine paragraph mark before its end-of-section character, rather than landing on the table's own row-ending mark. A `describe("writeDocContent inline pictures")` block covers a PNG and a JPEG image round-tripping their own raw bytes/format/size, a paragraph's own text splitting around an inline picture into separate blocks, more than one picture landing at distinct offsets in the same `"Data"` stream, an unwritable format (`svg`) throwing `DocUnsupportedError`, and no `"Data"` stream at all being written for a picture-free document. Neither of these two write-side additions has yet been checked against a real, independent [MS-DOC] implementation the way [Tables](#tables) and [Numbering definitions](#numbering-definitions) have -- that is the next thing worth doing here, mirroring how each of those was itself first verified by this package's own round trip alone.
 
-There is no real-world conformance corpus on the read side, and the write side inherits the same gap for the same reason: the tests prove this package matches the published specification, which is not the same as proving it matches what Word itself reads or writes between 1997 and 2007. Anyone extending this package should treat a corpus as the next thing worth building.
+A real-producer corpus layer exists: `pnpm test:corpus` runs the gitignored `test/corpus/` suite against LibreOffice-produced Word 97 documents generated by `scripts/generate-corpus.mjs` (flat-ODT and packaged-ODT sources spanning paragraphs, runs, headings, lists, tables with merges, images, and sections, converted headlessly through Writer's own export filter). That corpus already paid for itself once: it exposed that the reader's inline-picture locator assumed Word's OfficeArt wrapper nesting, which LibreOffice does not follow, and the locator now validates blip candidates by their payload file signatures instead. What a LibreOffice-produced corpus does not prove remains stated plainly: it is a real application's Word 97 spelling, not Word 1997-2007's own.
 
 ## Specification
 

packages/doc-codec/package.json

@@ -65,7 +65,9 @@
     "_test:workers": "vitest run --config vitest.workers.config.ts",
     "test:smoke": "turbo run _test:smoke",
     "_test:smoke": "vitest run --project smoke",
-    "prepare": "husky"
+    "prepare": "husky",
+    "test:corpus": "turbo run _test:corpus",
+    "_test:corpus": "vitest run --project corpus"
   },
   "keywords": [
     "doc",