diff --git a/packages/doc-codec/.gitignore b/packages/doc-codec/.gitignore index 5903e55d5..67ccec38c 100644 --- a/packages/doc-codec/.gitignore +++ b/packages/doc-codec/.gitignore @@ -5,3 +5,4 @@ dist/ coverage/ .eslintcache .turbo/ +test/corpus/ diff --git a/packages/doc-codec/README.md b/packages/doc-codec/README.md index 5d9d593dc..80f040078 100644 --- a/packages/doc-codec/README.md +++ b/packages/doc-codec/README.md @@ -44,7 +44,7 @@ Built and shipped, on the write side — see [Writing](#writing) for the full sc | **Table and numbering style formatting** | `STD.grLPUpxSw` is resolved only for paragraph and character styles (`stk` 1/2, see the row above) -- a table style's `StkTableGRLPUPX` (its own `TAPX` plus `CNFC`-conditional per-region formatting, [MS-DOC] 2.4.6.6's own table-specific steps 6-9) and a numbering style's `StkListGRLPUPX` are neither read nor written. A table applying a named table style, or a paragraph using a numbering style rather than direct list formatting, reports none of the formatting either would supply. | | **Subdocuments, on write; text boxes, on either side** | Footnotes, endnotes, comments, and headers/footers are now read (see the three read-side bullets above); none is written — `writeDocContent` still only ever produces a main document with `ccpFtn`/`ccpHdd`/`ccpAtn`/`ccpEdn` all zero, and a `ContentDocument` carries nowhere on its own `ContentSection.blocks` to state a header/footer's own content distinctly from the body's, which a real inverse would need. Text boxes are not read or written in either direction: `PlcfTxbxTxt` divides its own subdocument into stories the identical way the note-text plexes above do, but a text box's own anchor rides the drawn-object mechanism ([Images](#images)'s own scope note), which this package does not resolve. | | **Extended and user-defined document properties** | `title`/`subject`/`author`/`keywords`/`createdIso`/`modifiedIso` are read from and written to a `"\x05SummaryInformation"` stream when present (see [Metadata](#metadata)); the sibling `"\x05DocumentSummaryInformation"` stream (company, manager, and custom user-defined properties) is not read or written at all. | -| **Encryption** | RC4-encrypted ([MS-DOC] 2.2.6.2, ExaDev/documents.js#1113) and XOR-obfuscated (2.2.6.1, #922) documents are both decrypted given a password -- `readDocContent`/`readDocStreams` take an optional `password`, verified against the Table stream's own EncryptionHeader for RC4, or against `FibBase.lKey` directly for XOR obfuscation (see [Encryption](#encryption) below). RC4 CryptoAPI (a different, newer EncryptionHeader shape) still refuses with a `DocUnsupportedError` rather than being read as plaintext or misread as the wrong scheme. `writeDocContent` never encrypts. | +| **Encryption** | RC4-encrypted ([MS-DOC] 2.2.6.2, ExaDev/documents.js#1113) and XOR-obfuscated (2.2.6.1, #922) documents are both decrypted given a password -- `readDocContent`/`readDocStreams` take an optional `password`, verified against the Table stream's own EncryptionHeader for RC4, or against `FibBase.lKey` directly for XOR obfuscation (see [Encryption](#encryption) below). RC4 CryptoAPI (a different, newer EncryptionHeader shape) still refuses with a `DocUnsupportedError` rather than being read as plaintext or misread as the wrong scheme. `writeDocContent` never encrypts. | | **`sprmPHugePapx` / `sprmPTableProps`** | Paragraph properties stored indirectly in the Data stream are not followed, so such a paragraph reads with fewer properties than it states. [MS-DOC] 2.4.3's own Overview of Tables text names `sprmPTableProps` as a real, legal alternative to `sprmTDefTable` some applications process — but a real producer's row mark is not shown to prefer it: a genuine LibreOffice-authored `.doc` table's own row mark states its TAP through the identical direct `sprmTDefTable` this package's reader and writer already use (confirmed by parsing a LibreOffice 26.2.5.2-authored table's raw `PapxFkp` bytes; see [ExaDev/documents.js#892](https://github.com/ExaDev/documents.js/issues/892)), matching 2.4.3's own compatibility guidance ("An application SHOULD use sprmTDefTable to define table cells for applications that do not process sprmPTableProps"). `writeDocContent` never writes an indirect Papx. | | **Hyperlinks and fields** | `ContentRun.hyperlink`, footnote/comment/annotation references, and every other field or anchor character are read as plain text or dropped (see [What is converted](#what-is-converted)) and are not written. | | **Every FIB field beyond what this package's own reader needs** | `writeDocContent` populates only the fc/lcb pairs its own reader consults (the style sheet, the two property bin tables, the Clx, the font table). Roughly 140 other `FibRgFcLcb97` pairs — `SttbfAssoc`, `Dop`, the printer-driver structures among them — are left zero, which is the format's own "undefined, MUST be ignored" contract for most of them, but not a certification that every third-party [MS-DOC] reader accepts the result; see `fib/write.ts`'s own note. | @@ -345,7 +345,7 @@ The writer is verified the opposite way: `src/write.test.ts` reads every documen A `describe("writeDocContent multiple sections")` block (`ExaDev/documents.js#971`) covers two and three sections each round-tripping their own distinct page size and margins independently, and a non-final section whose own last block is a table still closing on a genuine paragraph mark before its end-of-section character, rather than landing on the table's own row-ending mark. A `describe("writeDocContent inline pictures")` block covers a PNG and a JPEG image round-tripping their own raw bytes/format/size, a paragraph's own text splitting around an inline picture into separate blocks, more than one picture landing at distinct offsets in the same `"Data"` stream, an unwritable format (`svg`) throwing `DocUnsupportedError`, and no `"Data"` stream at all being written for a picture-free document. Neither of these two write-side additions has yet been checked against a real, independent [MS-DOC] implementation the way [Tables](#tables) and [Numbering definitions](#numbering-definitions) have -- that is the next thing worth doing here, mirroring how each of those was itself first verified by this package's own round trip alone. -There is no real-world conformance corpus on the read side, and the write side inherits the same gap for the same reason: the tests prove this package matches the published specification, which is not the same as proving it matches what Word itself reads or writes between 1997 and 2007. Anyone extending this package should treat a corpus as the next thing worth building. +A real-producer corpus layer exists: `pnpm test:corpus` runs the gitignored `test/corpus/` suite against LibreOffice-produced Word 97 documents generated by `scripts/generate-corpus.mjs` (flat-ODT and packaged-ODT sources spanning paragraphs, runs, headings, lists, tables with merges, images, and sections, converted headlessly through Writer's own export filter). That corpus already paid for itself once: it exposed that the reader's inline-picture locator assumed Word's OfficeArt wrapper nesting, which LibreOffice does not follow, and the locator now validates blip candidates by their payload file signatures instead. What a LibreOffice-produced corpus does not prove remains stated plainly: it is a real application's Word 97 spelling, not Word 1997-2007's own. ## Specification diff --git a/packages/doc-codec/package.json b/packages/doc-codec/package.json index fdc8e784a..d793e7aef 100644 --- a/packages/doc-codec/package.json +++ b/packages/doc-codec/package.json @@ -65,7 +65,9 @@ "_test:workers": "vitest run --config vitest.workers.config.ts", "test:smoke": "turbo run _test:smoke", "_test:smoke": "vitest run --project smoke", - "prepare": "husky" + "prepare": "husky", + "test:corpus": "turbo run _test:corpus", + "_test:corpus": "vitest run --project corpus" }, "keywords": [ "doc", diff --git a/packages/doc-codec/scripts/generate-corpus.mjs b/packages/doc-codec/scripts/generate-corpus.mjs new file mode 100644 index 000000000..b7aa5cc47 --- /dev/null +++ b/packages/doc-codec/scripts/generate-corpus.mjs @@ -0,0 +1,398 @@ +#!/usr/bin/env node +// Generates this package's gitignored real-producer .doc corpus under test/corpus/, ready for `pnpm test:corpus`. The producer is LibreOffice's own MS Word 97 export filter: each fixture is authored as a flat-ODT source (construct-spanning, the constructs this package's reader documents itself as reading) and converted headlessly through `soffice --convert-to doc`, so every corpus file is genuine application output rather than hand-built bytes. Run from the package root: `node scripts/generate-corpus.mjs` (requires soffice at /opt/homebrew/bin/soffice or SOFFICE in the environment; overwrites test/corpus/ wholesale). The generated corpus.test.ts carries each fixture's expectations -- authored beside the fixture's construction here, so the generator is the single source of truth for both. What this corpus is and is not is stated in the README: a real producer's Word 97 spelling, not Word 1997-2007 itself. +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const packageRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); +const outDir = join(packageRoot, "test", "corpus"); +const soffice = process.env.SOFFICE ?? "/opt/homebrew/bin/soffice"; + +// A 2x2 solid-red PNG (CRC-correct -- LibreOffice's libpng rejects a bad IHDR CRC by silently dropping the picture, which the first draft of this constant did). +const TINY_PNG = + "iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAIAAAD91JpzAAAAEElEQVR4nGP4z8AARAwQCgAf7gP9i18U1AAAAABJRU5ErkJggg=="; + +const NS = + 'xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" ' + + 'xmlns:style="urn:oasis:names:tc:opendocument:xmlns:style:1.0" ' + + 'xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" ' + + 'xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0" ' + + 'xmlns:draw="urn:oasis:names:tc:opendocument:xmlns:drawing:1.0" ' + + 'xmlns:fo="urn:oasis:names:tc:opendocument:xmlns:xsl-fo-compatible:1.0" ' + + 'xmlns:svg="urn:oasis:names:tc:opendocument:xmlns:svg-compatible:1.0" ' + + 'xmlns:xlink="http://www.w3.org/1999/xlink" ' + + 'xmlns:dc="http://purl.org/dc/elements/1.1/"'; + +const FIXTURES = [ + { + name: "plain", + body: "Corpus plain paragraph.", + expect: { + text: ["Corpus plain paragraph."], + }, + }, + { + name: "runs", + body: + "" + + 'bold run and ' + + 'italic run and ' + + 'underlined run' + + "", + expect: { + runs: [ + { text: "bold run", bold: true }, + { text: "italic run", italic: true }, + { text: "underlined run", underline: true }, + ], + }, + }, + { + name: "headings", + body: + 'Chapter One' + + 'Section A' + + "Body under the section.", + expect: { + headings: [ + { text: "Chapter One", level: 1 }, + { text: "Section A", level: 2 }, + ], + }, + }, + { + name: "lists", + body: + 'bullet onebullet two' + + 'step onestep two', + expect: { + listItems: ["bullet one", "bullet two", "step one", "step two"], + }, + }, + { + name: "table", + body: + '' + + "" + + "" + + "alpha" + + 'wide' + + "" + + "" + + "" + + "beta" + + "gamma" + + "delta" + + "" + + "", + expect: { + tables: [ + { + rows: 2, + cells: ["alpha", "wide", "beta", "gamma", "delta"], + colSpanAt: { row: 0, cell: 1, span: 2 }, + }, + ], + }, + }, + { + name: "image", + // Writer's import drops a draw:image carrying office:binary-data from a FLAT source (Calc accepts the same spelling), so this fixture's source is a minimal packaged .odt with the picture as a real media entry -- see packageOdt below. + packagedMedia: { name: "Pictures/img.png", base64: TINY_PNG }, + body: + "Before the picture." + + "" + + '' + + "" + + "After the picture.", + expect: { + text: ["Before the picture.", "After the picture."], + images: 1, + }, + }, + { + name: "sections", + body: + "First section body." + + 'Second section body.', + expect: { + text: ["First section body.", "Second section body."], + }, + }, + { + name: "mixed", + body: + 'Mixed Chapter' + + 'Weighted lead-in.' + + 'mixed bullet' + + "Tail paragraph.", + expect: { + headings: [{ text: "Mixed Chapter", level: 1 }], + runs: [{ text: "Weighted", bold: true }], + listItems: ["mixed bullet"], + text: [ + "Mixed Chapter", + "Weighted lead-in.", + "mixed bullet", + "Tail paragraph.", + ], + }, + }, +]; + +const AUTOSTYLES = + " " + + '' + + '' + + '' + + '' + + '' + + " "; + +// CRC-32 (IEEE) over one buffer, the zip central-directory spelling needs. +function crc32(bytes) { + let c = ~0; + for (const byte of bytes) { + c ^= byte; + for (let k = 0; k < 8; k++) { + c = (c >>> 1) ^ (0xedb88320 & -(c & 1)); + } + } + return ~c >>> 0; +} + +// A minimal STORED (uncompressed) .odt package: mimetype first with no extra fields, then content.xml, the manifest naming the media entry, and the media bytes. Just enough package for Writer to load a picture a flat source cannot carry. +function packageOdt(contentXml, mediaName, mediaBytes) { + const enc = new TextEncoder(); + const entries = [ + { + name: "mimetype", + data: enc.encode("application/vnd.oasis.opendocument.text"), + }, + { name: "content.xml", data: enc.encode(contentXml) }, + { + name: "META-INF/manifest.xml", + data: enc.encode( + '\n' + + '' + + '' + + '' + + '', + ), + }, + { name: mediaName, data: mediaBytes }, + ]; + const chunks = []; + const central = []; + let offset = 0; + for (const entry of entries) { + const nameBytes = enc.encode(entry.name); + const crc = crc32(entry.data); + const localHeader = new DataView(new ArrayBuffer(30)); + localHeader.setUint32(0, 0x04034b50, true); + localHeader.setUint16(4, 20, true); + localHeader.setUint16(6, 0, true); + localHeader.setUint16(8, 0, true); // stored + localHeader.setUint16(10, 0, true); + localHeader.setUint16(12, 0, true); + localHeader.setUint32(14, crc, true); + localHeader.setUint32(18, entry.data.length, true); + localHeader.setUint32(22, entry.data.length, true); + localHeader.setUint16(26, nameBytes.length, true); + localHeader.setUint16(28, 0, true); + chunks.push(new Uint8Array(localHeader.buffer), nameBytes, entry.data); + central.push({ nameBytes, crc, size: entry.data.length, offset }); + offset += 30 + nameBytes.length + entry.data.length; + } + const centralStart = offset; + for (const e of central) { + const h = new DataView(new ArrayBuffer(46)); + h.setUint32(0, 0x02014b50, true); + h.setUint16(4, 20, true); + h.setUint16(6, 20, true); + h.setUint32(16, e.crc, true); + h.setUint32(20, e.size, true); + h.setUint32(24, e.size, true); + h.setUint16(28, e.nameBytes.length, true); + h.setUint32(42, e.offset, true); + chunks.push(new Uint8Array(h.buffer), e.nameBytes); + offset += 46 + e.nameBytes.length; + } + const end = new DataView(new ArrayBuffer(22)); + end.setUint32(0, 0x06054b50, true); + end.setUint16(8, central.length, true); + end.setUint16(10, central.length, true); + end.setUint32(12, offset - centralStart, true); + end.setUint32(16, centralStart, true); + chunks.push(new Uint8Array(end.buffer)); + const total = chunks.reduce((n, c) => n + c.length, 0); + const out = new Uint8Array(total); + let at = 0; + for (const c of chunks) { + out.set(c, at); + at += c.length; + } + return out; +} + +function fodt(body) { + return ( + '\n\n' + + AUTOSTYLES + + "\n " + + body + + "\n" + ); +} + +// The corpus harness: reads each converted .doc through this package's own reader and asserts the manifest's expectations against the recovered ContentDocument. Regenerated by the script alongside the fixtures -- the whole test/corpus/ layer is local-only by the family's convention, and the script is the committed source of truth. +const CORPUS_TEST = `import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { readDocContent } from "../../src/read"; + +const MANIFEST = JSON.parse( + readFileSync(join(import.meta.dirname, "manifest.json"), "utf8"), +) as readonly { + file: string; + expect: Record; +}[]; + +describe("doc corpus (LibreOffice-produced Word 97)", () => { + for (const { file, expect: e } of MANIFEST) { + it(file + " reads back what its flat-ODT source authored", () => { + const bytes = new Uint8Array( + readFileSync(join(import.meta.dirname, file)), + ); + const document = readDocContent(bytes); + if (document.kind !== "wordprocessing") { + throw new Error("expected a wordprocessing document"); + } + const paragraphs = document.sections.flatMap((s) => + s.blocks.filter((b) => b.kind === "paragraph"), + ); + const blocks = document.sections.flatMap((s) => s.blocks); + const text = blocks + .flatMap((b) => (b.kind === "paragraph" ? b.runs.map((r) => r.text) : [])) + .join(" ") + .replace(/\\s+/g, " ") + .trim(); + if (Array.isArray(e.text)) { + for (const needle of e.text as string[]) { + expect(text).toContain(needle); + } + } + if (Array.isArray(e.runs)) { + const all = paragraphs.flatMap((b) => (b.kind === "paragraph" ? b.runs : [])); + for (const want of e.runs as { text: string; bold?: boolean; italic?: boolean; underline?: boolean }[]) { + const run = all.find((r) => r.text === want.text); + expect(run, "run " + want.text + " recovered").toBeDefined(); + if (want.bold !== undefined) expect(run!.bold ?? false).toBe(want.bold); + if (want.italic !== undefined) expect(run!.italic ?? false).toBe(want.italic); + if (want.underline !== undefined) expect(run!.underline ?? false).toBe(want.underline); + } + } + if (Array.isArray(e.headings)) { + const all = blocks.filter( + (b): b is Extract => b.kind === "paragraph", + ); + for (const want of e.headings as { text: string; level: number }[]) { + const heading = all.find( + (p) => p.runs.map((r) => r.text).join("") === want.text, + ); + expect(heading, "heading " + want.text).toBeDefined(); + expect(heading!.headingLevel).toBe(want.level); + } + } + if (Array.isArray(e.listItems)) { + const all = blocks.filter( + (b): b is Extract => b.kind === "paragraph", + ); + for (const item of e.listItems as string[]) { + const para = all.find((p) => p.runs.map((r) => r.text).join("") === item); + expect(para, "list item " + item).toBeDefined(); + expect(para!.list).toBeDefined(); + } + } + if (Array.isArray(e.tables)) { + const tables = blocks.filter( + (b): b is Extract => b.kind === "table", + ); + expect(tables).toHaveLength((e.tables as unknown[]).length); + for (const [ti, want] of (e.tables as { rows: number; cells: string[]; colSpanAt?: { row: number; cell: number; span: number } }[]).entries()) { + const table = tables[ti]!; + expect(table.rows).toHaveLength(want.rows); + const cells = table.rows.flatMap((r) => r.cells); + expect(cells.map((c) => c.blocks.map((b) => (b.kind === "paragraph" ? b.runs.map((r) => r.text).join("") : "")).join(""))).toEqual(want.cells); + if (want.colSpanAt !== undefined) { + expect(table.rows[want.colSpanAt.row]!.cells[want.colSpanAt.cell]!.colSpan).toBe(want.colSpanAt.span); + } + } + } + if (typeof e.images === "number") { + const images = blocks.filter( + (b) => b.kind === "image", + ); + expect(images).toHaveLength(e.images); + } + }); + } +}); +`; + +rmSync(outDir, { recursive: true, force: true }); +mkdirSync(outDir, { recursive: true }); +const staging = join(outDir, ".staging"); +mkdirSync(staging, { recursive: true }); + +const manifest = []; +for (const fixture of FIXTURES) { + const src = join( + staging, + fixture.name + (fixture.packagedMedia === undefined ? ".fodt" : ".odt"), + ); + if (fixture.packagedMedia === undefined) { + writeFileSync(src, fodt(fixture.body)); + } else { + // The packaged source's content.xml uses the document-content root and carries the manifest-named media the frame points at. + const contentXml = + '\n' + + "' + + fixture.body + + ""; + const media = Buffer.from(fixture.packagedMedia.base64, "base64"); + writeFileSync( + src, + packageOdt(contentXml, fixture.packagedMedia.name, media), + ); + } + execFileSync( + soffice, + [ + "--headless", + "--norestore", + "--convert-to", + "doc", + "--outdir", + staging, + src, + ], + { stdio: "pipe" }, + ); + rmSync(src, { force: true }); + const converted = join(staging, fixture.name + ".doc"); + writeFileSync(join(outDir, fixture.name + ".doc"), readFileSync(converted)); + manifest.push({ file: fixture.name + ".doc", expect: fixture.expect }); +} +rmSync(staging, { recursive: true, force: true }); + +writeFileSync(join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2)); +writeFileSync(join(outDir, "corpus.test.ts"), CORPUS_TEST); +console.log("doc corpus: " + manifest.length + " fixtures under test/corpus/"); diff --git a/packages/doc-codec/src/pictures.test.ts b/packages/doc-codec/src/pictures.test.ts index f291e5cf3..6eae77dc9 100644 --- a/packages/doc-codec/src/pictures.test.ts +++ b/packages/doc-codec/src/pictures.test.ts @@ -7,7 +7,8 @@ import { INLINE_PICTURE } from "./text/special"; describe("readDocContent inline pictures", () => { it("reads an inline picture's own PNG bytes and its size from PICMID's dxaGoal/dyaGoal", () => { - const pngBytes = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]); + // Signature-led, as every real PNG blip's payload begins ([MS-ODRAW]'s OfficeArtBlipPNG carries raw file bytes); the reader validates the signature when locating the blip. + const pngBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 1, 2, 3, 4, 5]); const picLocation = 0x40; const { dataStreamBytes, picLocationGrpprl } = buildInlinePictureBytes( picLocation, @@ -46,7 +47,7 @@ describe("readDocContent inline pictures", () => { }); it("splits a paragraph carrying real text around an inline picture into separate blocks", () => { - const pngBytes = new Uint8Array([9, 9, 9]); + const pngBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 9, 9, 9]); const picLocation = 0x40; const { dataStreamBytes, picLocationGrpprl } = buildInlinePictureBytes( picLocation, diff --git a/packages/doc-codec/src/pictures.ts b/packages/doc-codec/src/pictures.ts index 73099f527..a0e7cc8d1 100644 --- a/packages/doc-codec/src/pictures.ts +++ b/packages/doc-codec/src/pictures.ts @@ -8,9 +8,9 @@ import { slice, } from "./bytes"; -// Inline pictures, [MS-DOC] "Pictures": a picture character (U+0001, sprmCFSpec applied) names its own data through sprmCPicLocation, a signed 32-bit offset into the Data stream where a PICFAndOfficeArtData structure lives -- a PICF (68 bytes: type/size/border information) followed, for every producer this reader has seen, by a real OfficeArtInlineSpContainer ([MS-ODRAW] 2.2.15) regardless of PICF.mfpf.mm's own value, since [MS-DOC] states the `picture` field itself as that container's type. There is no simpler, non-OfficeArt path even for the plainest bitmap. +// Inline pictures, [MS-DOC] "Pictures": a picture character (U+0001, sprmCFSpec applied) names its own data through sprmCPicLocation, a signed 32-bit offset into the Data stream where a PICFAndOfficeArtData structure lives -- a PICF (68 bytes: type/size/border information) followed by the picture's OfficeArt container chain. Word wraps the blip in an OfficeArtInlineSpContainer ([MS-ODRAW] 2.2.15); LibreOffice (verified against a real LibreOffice-produced corpus file) wraps it in a SpgrContainer with a property table and no InlineSp wrapper at all -- so the reader walks container headers forward from PICF's end until the blip's own record type appears, tolerating either producer's wrapper shape without looking inside any of them. There is no simpler, non-OfficeArt path even for the plainest bitmap. // -// This reads exactly as much of that container as the common case needs: OfficeArtInlineSpContainer.shape (an OfficeArtSpContainer, [MS-ODRAW] 2.2.14) is skipped whole by its own record header's recLen, and the first entry of `rgfb` immediately after it -- an OfficeArtBStoreContainerFileBlock, in practice a single OfficeArtBlip record for one inline picture with no separate blip-store indirection -- is read directly. Only the two raster formats document-schema.js's ContentImageBlock can hold losslessly (OfficeArtBlipJPEG 0xF01D, OfficeArtBlipPNG 0xF01E) are decoded; every other blip kind (WMF/EMF/PICT metafiles, a raw DIB with no format this schema names, TIFF) is a genuinely different structure -- a metafile blip carries a further OfficeArtMetafileHeader and, for WMF/EMF, DEFLATE-compressed payload bytes; a DIB has no ContentImageBlock format token to hold it under at all without re-encoding pixels this package has no image codec to perform -- so those return undefined here rather than being mis-decoded, the identical "genuinely unimplemented, not approximated" convention the rest of this package's own scope table already follows for floating drawn objects (PlcfSpa/OfficeArt shapes generally) and text boxes, which this module does not attempt at all. +// This reads exactly as much of that chain as the common case needs: the validated blip record the locator lands on is read directly -- in practice a single OfficeArtBlip record for one inline picture with no separate blip-store indirection -- and every wrapper container before it is skipped without being parsed at all. Only the two raster formats document-schema.js's ContentImageBlock can hold losslessly (OfficeArtBlipJPEG 0xF01D, OfficeArtBlipPNG 0xF01E) are decoded; every other blip kind (WMF/EMF/PICT metafiles, a raw DIB with no format this schema names, TIFF) is a genuinely different structure -- a metafile blip carries a further OfficeArtMetafileHeader and, for WMF/EMF, DEFLATE-compressed payload bytes; a DIB has no ContentImageBlock format token to hold it under at all without re-encoding pixels this package has no image codec to perform -- so those return undefined here rather than being mis-decoded, the identical "genuinely unimplemented, not approximated" convention the rest of this package's own scope table already follows for floating drawn objects (PlcfSpa/OfficeArt shapes generally) and text boxes, which this module does not attempt at all. const PICF_SIZE = 68; const PICF_MM_OFFSET = 6; @@ -74,11 +74,12 @@ export function readInlinePicture( cursor += 1 + cchPicName; } - // OfficeArtInlineSpContainer.shape: an OfficeArtSpContainer, skipped whole by its own record header's recLen -- this reader has no need to look inside it (the shape's own fill/line/position properties, not the picture's own bytes). - const shapeHeader = readRecordHeader(dataStream, cursor); - cursor += RECORD_HEADER_SIZE + shapeHeader.recLen; - - const blipHeader = readRecordHeader(dataStream, cursor); + // Locating the blip: the containers between PICF and the blip are wrapper shapes this reader has no need to look inside, and producers disagree on the nesting -- Word writes InlineSpContainer > SpContainer > blip, while LibreOffice (confirmed against a real LibreOffice-produced .doc corpus file, 2026-09-10) emits a chain whose container lengths do not walk to the blip (its property-table record's recLen spans past the blip entirely), so header-walking mis-parses it. The robust spelling-independent locator: scan forward from PICF's end for a record header whose type is a known blip, whose instance names a known rgbUid count, whose length stays inside the Data stream, and whose payload actually begins with that format's own file signature -- a validated blip, not merely a well-formed header. The signature check is what makes a false positive on wrapper bytes effectively impossible: no container prefix preceding a real blip starts with a PNG or JPEG signature at exactly the uid-and-tag-derived offset. + const found = findBlipRecord(dataStream, cursor); + if (found === undefined) { + return undefined; + } + const { header: blipHeader, offset: blipOffset } = found; const format = blipFormat(blipHeader.recType); if (format === undefined) return undefined; @@ -89,7 +90,8 @@ export function readInlinePicture( : undefined; if (uidBytes === undefined) return undefined; - const blipDataStart = cursor + RECORD_HEADER_SIZE + uidBytes + BLIP_TAG_SIZE; + const blipDataStart = + blipOffset + RECORD_HEADER_SIZE + uidBytes + BLIP_TAG_SIZE; const blipDataLength = blipHeader.recLen - uidBytes - BLIP_TAG_SIZE; const blipBytes = slice( dataStream, @@ -117,3 +119,55 @@ function blipFormat(recType: number): "jpeg" | "png" | undefined { return undefined; } } + +// The PNG and JPEG file signatures, the one-byte-prefix form OfficeArtBlip carries them under (rgbUid, then the one-byte tag, then raw file bytes). +const PNG_SIGNATURE = [0x89, 0x50, 0x4e, 0x47]; +const JPEG_SIGNATURE = [0xff, 0xd8]; + +function payloadHasSignature( + data: Uint8Array, + start: number, + signature: readonly number[], +): boolean { + for (const [i, byte] of signature.entries()) { + if (data[start + i] !== byte) { + return false; + } + } + return true; +} + +/** A record header at a known offset, the validated-blip scan's answer. */ +interface FoundBlip { + readonly header: RecordHeader; + readonly offset: number; +} + +/** Scans forward from `from` for a validated blip record (see readInlinePicture's own locating note) -- every candidate header of a blip type must also carry a known rgbUid instance count, a length inside the stream, and payload bytes starting with its format's own file signature. */ +function findBlipRecord(data: Uint8Array, from: number): FoundBlip | undefined { + for (let at = from; at + RECORD_HEADER_SIZE <= data.length; at++) { + const header = readRecordHeader(data, at); + const format = blipFormat(header.recType); + if (format === undefined) { + continue; + } + const uidBytes = ONE_UID_INSTANCES.has(header.recInstance) + ? 16 + : TWO_UID_INSTANCES.has(header.recInstance) + ? 32 + : undefined; + if (uidBytes === undefined) { + continue; + } + const payloadStart = at + RECORD_HEADER_SIZE + uidBytes + BLIP_TAG_SIZE; + const signature = format === "png" ? PNG_SIGNATURE : JPEG_SIGNATURE; + if ( + header.recLen > uidBytes + BLIP_TAG_SIZE && + payloadStart + signature.length <= data.length && + payloadHasSignature(data, payloadStart, signature) + ) { + return { header, offset: at }; + } + } + return undefined; +} diff --git a/packages/doc-codec/src/write.test.ts b/packages/doc-codec/src/write.test.ts index f0a0f5435..58a11d45f 100644 --- a/packages/doc-codec/src/write.test.ts +++ b/packages/doc-codec/src/write.test.ts @@ -580,7 +580,7 @@ describe("writeDocContent inline pictures", () => { { kind: "image", format: "png", - base64: base64Of([1, 2, 3]), + base64: base64Of([0x89, 0x50, 0x4e, 0x47, 1, 2, 3]), widthPt: 10, heightPt: 10, }, @@ -600,14 +600,14 @@ describe("writeDocContent inline pictures", () => { { kind: "image", format: "png", - base64: base64Of([1, 1, 1]), + base64: base64Of([0x89, 0x50, 0x4e, 0x47, 1, 1, 1]), widthPt: 10, heightPt: 10, }, { kind: "image", format: "png", - base64: base64Of([2, 2, 2, 2]), + base64: base64Of([0x89, 0x50, 0x4e, 0x47, 2, 2, 2, 2]), widthPt: 20, heightPt: 20, }, @@ -621,10 +621,10 @@ describe("writeDocContent inline pictures", () => { } expect( Array.from(atob(first.base64), (char) => char.charCodeAt(0)), - ).toEqual([1, 1, 1]); + ).toEqual([0x89, 0x50, 0x4e, 0x47, 1, 1, 1]); expect( Array.from(atob(second.base64), (char) => char.charCodeAt(0)), - ).toEqual([2, 2, 2, 2]); + ).toEqual([0x89, 0x50, 0x4e, 0x47, 2, 2, 2, 2]); }); it("refuses an image format it cannot write, such as svg", () => { diff --git a/packages/doc-codec/vitest.config.ts b/packages/doc-codec/vitest.config.ts index d3c393ed3..e9ff398b4 100644 --- a/packages/doc-codec/vitest.config.ts +++ b/packages/doc-codec/vitest.config.ts @@ -1,6 +1,6 @@ import { defineConfig } from "vitest/config"; -// Two named projects in one config, filtered by --project in package.json's scripts: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. There is no "corpus" project yet: a real-world .doc conformance corpus needs licensable sample documents, which this package does not have. +// Two named projects in one config, filtered by --project in package.json's scripts: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. A "corpus" project (test/corpus/**/*.test.ts) holds the optional, gitignored real-producer conformance layer -- LibreOffice-produced Word 97 documents generated into test/corpus/ by scripts/generate-corpus.mjs, run only by pnpm test:corpus and never part of pnpm test. export default defineConfig({ test: { // Vitest resolves coverage once for the whole run from this root config, not per project, so it cannot live inside the 'unit' project's own test block; pnpm test:coverage scopes what actually gets measured by filtering to --project unit, which never imports the smoke suite. @@ -13,6 +13,7 @@ export default defineConfig({ projects: [ { test: { name: "unit", include: ["src/**/*.test.ts"] } }, { test: { name: "smoke", include: ["test/smoke.test.mjs"] } }, + { test: { name: "corpus", include: ["test/corpus/**/*.test.ts"] } }, ], }, }); diff --git a/packages/document-compute.js/.gitignore b/packages/document-compute.js/.gitignore index 5903e55d5..67ccec38c 100644 --- a/packages/document-compute.js/.gitignore +++ b/packages/document-compute.js/.gitignore @@ -5,3 +5,4 @@ dist/ coverage/ .eslintcache .turbo/ +test/corpus/ diff --git a/packages/document-compute.js/README.md b/packages/document-compute.js/README.md index 96518a34d..bd23e6fc0 100644 --- a/packages/document-compute.js/README.md +++ b/packages/document-compute.js/README.md @@ -113,7 +113,7 @@ One thing #573 asks for was closed at adoption rather than built here: `Quantity Scoped to point-valued (`Quantity`) answers: every value this harness computes comes from evaluating a closed statement with no bindings, which `evaluate` cannot turn into an `Interval` (an `Interval` only ever arises by binding a symbol to one) — a genuinely interval-valued worked example (`0.87 <= cos(phi) <= 1`, #573's own illustration of interval arithmetic) has no representation in this "symbol = expression" equality grammar at all, since neither `MathExpression` nor `documents.js`'s LaTeX lowering has a compound-inequality-to-range reading, and is out of scope for this pass rather than silently mishandled. -`src/harness/corpus.test.ts` proves the whole pipeline end to end — markdown text through `markdown-codec`'s `$$` block recognition and `documents.js`'s `lowerMarkdownMath` (the "LaTeX lowering" #794 names as the natural source of worked examples) into this harness — against a small, hand-authored starter corpus. `markdown-codec` and `documents.js` are **devDependencies only**: both sit above this package in the family's own dependency order (see the monorepo root README's package table), so neither can be a runtime dependency here without a cycle. The harness itself is not test-only, though — `src/index.ts` exports it as real public API (`runWorkedExampleSequence`, `collectFormulas`, `runCorpus`, `formatCorpusReport`), the same as `evaluate`/`solveFor`; what's actually true is narrower: it has no runtime consumer anywhere else in the family yet, unlike `evaluate`/`solveFor` below it, which `document-mcp`'s `compute_formula` tool does depend on directly (see Conventions). A large real-world corpus (the issue's own stated differentiator at scale) is not included — gathering one is a data-curation task, not a code one — but is a straightforward local addition: point a `test/corpus/` directory (gitignored, matching `pdf-codec`'s own `test:corpus` convention) at real markdown documents with worked examples and feed `readMarkdownContent` → `lowerMarkdownMath` → `runCorpus` the same way `corpus.test.ts` does. +`src/harness/corpus.test.ts` proves the whole pipeline end to end — markdown text through `markdown-codec`'s `$$` block recognition and `documents.js`'s `lowerMarkdownMath` (the "LaTeX lowering" #794 names as the natural source of worked examples) into this harness — against a small, hand-authored starter corpus. `markdown-codec` and `documents.js` are **devDependencies only**: both sit above this package in the family's own dependency order (see the monorepo root README's package table), so neither can be a runtime dependency here without a cycle. The harness itself is not test-only, though — `src/index.ts` exports it as real public API (`runWorkedExampleSequence`, `collectFormulas`, `runCorpus`, `formatCorpusReport`), the same as `evaluate`/`solveFor`; what's actually true is narrower: it has no runtime consumer anywhere else in the family yet, unlike `evaluate`/`solveFor` below it, which `document-mcp`'s `compute_formula` tool does depend on directly (see Conventions). An at-scale corpus layer now exists: `pnpm test:corpus` runs the gitignored `test/corpus/` suite that `scripts/generate-corpus.mjs` regenerates — 300 deterministic worked-example documents over the mechanically-lowered arithmetic grammar (explicit \times/\frac/\sqrt compositions, never juxtaposition), each stated answer computed by the generator's own parallel evaluation — measuring 300/300 matched, 0 gaps, 0 unresolved, with the measured report written to `test/corpus/report.txt`. Its first run paid for itself immediately: every document with a NEGATIVE stated answer degraded its whole equality under the lowering's leading-minus-only unary reading, which now generalises past that (a minus after a relation or operator signs the following operand). Real textbook corpora remain a local addition on top: point the same `test/corpus/files/` directory at real markdown documents and the harness measures them identically. While building this harness's own fixtures, a real bug surfaced in `documents.js`'s LaTeX lowering: `F = m \times a` (the textbook-standard way to write almost any formula) lowers to `(F = m) \times a` rather than `F = (m \times a)`, because the lowering folds relational and arithmetic operators at the same precedence with no notion that `=` should bind loosest — filed as [ExaDev/documents.js#812](https://github.com/ExaDev/documents.js/issues/812). This package's own fixtures work around it with an explicit braced right-hand side (`F = {m \times a}`, which lowers correctly), since fixing the lowering itself is out of scope for this package. diff --git a/packages/document-compute.js/package.json b/packages/document-compute.js/package.json index c9fc25e0c..6240a7381 100644 --- a/packages/document-compute.js/package.json +++ b/packages/document-compute.js/package.json @@ -57,14 +57,16 @@ "_typecheck": "tsc -p tsconfig.json && tsc -p tsconfig.node.json", "_typecheck:attw": "attw --pack", "test": "turbo run _test", - "_test": "vitest run", - "_test:coverage": "vitest run --coverage", + "_test": "vitest run --project unit", + "_test:coverage": "vitest run --project unit --coverage", "test:mutation": "turbo run _test:mutation", "_test:mutation": "stryker run stryker.config.mjs", - "test:watch": "vitest", + "test:watch": "vitest --project unit", "test:workers": "turbo run _test:workers", "_test:workers": "vitest run --config vitest.workers.config.ts", - "prepare": "husky" + "prepare": "husky", + "test:corpus": "turbo run _test:corpus", + "_test:corpus": "vitest run --project corpus" }, "packageManager": "pnpm@11.6.0", "dependencies": { diff --git a/packages/document-compute.js/scripts/generate-corpus.mjs b/packages/document-compute.js/scripts/generate-corpus.mjs new file mode 100644 index 000000000..0417670e1 --- /dev/null +++ b/packages/document-compute.js/scripts/generate-corpus.mjs @@ -0,0 +1,178 @@ +#!/usr/bin/env node +// Generates this package's gitignored at-scale worked-example corpus under test/corpus/, ready for `pnpm test:corpus` -- the at-scale coverage measurement #956 names as the harness's own missing differentiator ("a small, hand-authored starter corpus" was all src/harness had). The generator authors N markdown documents, each a deterministic pseudo-random worked example from the mechanically-lowered arithmetic grammar (explicit \times/\frac/\sqrt/^ compositions -- never juxtaposition, which the lowering deliberately degrades to unparsed) with the stated answer computed by this same generator's parallel JS evaluation, rounded to 6 significant figures the way a textbook author rounds. A mismatch therefore names a genuine lowering/evaluation defect, not a fixture typo. Run from the package root: `node scripts/generate-corpus.mjs` (overwrites test/corpus/ wholesale). +import { mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const packageRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); +const outDir = join(packageRoot, "test", "corpus"); + +// Seeded LCG so the corpus is byte-identical across runs on the same seed. +function lcg(seed) { + let state = seed >>> 0; + return () => { + state = (state * 1664525 + 1013904223) >>> 0; + return state / 0x100000000; + }; +} + +const DOCUMENT_COUNT = 300; +// Single-letter symbol names: a digit-bearing token (r0) lowers to an unparsed node and a multi-letter token (qb) to juxtaposition -- only a single letter is the plain symbol the worked-example grammar matches on. Each document draws its own four distinct letters from a rotating window of the alphabet, so a document's givens and its target never collide. + +// The value pool: decimals and small integers, kept positive where sqrt is involved downstream. +function pickValue(random) { + const kind = Math.floor(random() * 3); + if (kind === 0) return Math.floor(random() * 9) + 1; + if (kind === 1) return Math.round((random() * 9 + 1) * 100) / 100; + return Math.round((random() * 89 + 10) * 10) / 1000; +} + +// One expression node: { latex (with symbol leaves), js (a JS expression string over the symbol names) }. +function buildExpression(random, symbols) { + const leaf = () => { + const s = symbols[Math.floor(random() * symbols.length)]; + return { latex: s, js: s }; + }; + const build = (depth) => { + if (depth === 0) return leaf(); + const kind = Math.floor(random() * 4); + const a = build(depth - 1); + const b = build(depth - 1); + if (kind === 0) + return { latex: `{${a.latex} + ${b.latex}}`, js: `(${a.js} + ${b.js})` }; + if (kind === 1) + return { + latex: `{${a.latex} - ${b.latex}}`, + js: `(${a.js} - ${b.js})`, + }; + if (kind === 2) + return { + latex: `{${a.latex} \\times ${b.latex}}`, + js: `(${a.js} * ${b.js})`, + }; + return { + latex: `\\frac{${a.latex}}{${b.latex}}`, + js: `(${a.js} / ${b.js})`, + }; + }; + let node = build(1 + Math.floor(random() * 2)); + // sqrt only over subexpressions the parallel JS evaluation proves positive (the generator evaluates bindings later, so guard structurally: wrap only additions and multiplications of positive leaves). + if (random() < 0.25 && node.latex.includes("+")) { + node = { latex: `\\sqrt{${node.latex}}`, js: `Math.sqrt(${node.js})` }; + } + return node; +} + +// Round to 6 significant figures, the textbook-author rounding the harness's own 1e-3 relative tolerance is built for. +function roundSig(value) { + if (value === 0) return 0; + const magnitude = Math.floor(Math.log10(Math.abs(value))); + const digits = 6 - 1 - magnitude; + const factor = 10 ** digits; + return Math.round(value * factor) / factor; +} + +function mathBlock(latex) { + return `$$\n${latex}\n$$`; +} + +const documents = []; +for (let i = 0; i < DOCUMENT_COUNT; i++) { + const random = lcg(0x5eed0000 + i); + // Two or three given symbols, dimensionless -- plain letter-digit names only, never underscores: an underscore is LaTeX's subscript marker, and a subscripted leaf lowers to a scripted symbol identity rather than the plain binding the worked-example grammar matches on. + const givenCount = 2 + Math.floor(random() * 2); + const symbols = []; + const givenLines = []; + for (let g = 0; g < givenCount; g++) { + const name = String.fromCharCode(97 + ((i * 4 + g) % 26)); + const value = pickValue(random); + symbols.push(name); + givenLines.push(`${name} = ${roundSig(value)}`); + } + const target = String.fromCharCode(65 + ((i * 4 + 3) % 26)); + const expression = buildExpression(random, symbols); + const env = {}; + symbols.forEach((name, g) => { + env[name] = parseFloat(givenLines[g].split("= ")[1]); + }); + const stated = roundSig( + new Function(...symbols, `return ${expression.js};`)( + ...symbols.map((s) => env[s]), + ), + ); + if (!Number.isFinite(stated)) { + // Division by zero can arise from the fraction branch; replace this document with a pure multiplication example so every fixture has a finite stated answer (a divide-by-zero corpus belongs to the unit suite, which already covers DivisionByZeroError). + const product = symbols.map((s) => s).join(" \\times "); + const value = symbols.reduce((acc, s) => acc * env[s], 1); + documents.push({ + name: `doc${i}.md`, + markdown: [ + mathBlock(`${target} = {${product}}`), + ...givenLines.map(mathBlock), + mathBlock(`${target} = ${roundSig(value)}`), + ].join("\n\n"), + }); + continue; + } + documents.push({ + name: `doc${i}.md`, + markdown: [ + mathBlock(`${target} = ${expression.latex}`), + ...givenLines.map(mathBlock), + mathBlock(`${target} = ${stated}`), + ].join("\n\n"), + }); +} + +rmSync(outDir, { recursive: true, force: true }); +mkdirSync(join(outDir, "files"), { recursive: true }); +for (const document of documents) { + writeFileSync(join(outDir, "files", document.name), document.markdown); +} + +// The corpus harness: lowers each markdown document through the identical pipeline the starter corpus uses and runs the at-scale measurement, printing formatCorpusReport's own text for the console. Regenerated by the script alongside the fixtures -- the whole test/corpus/ layer is local-only by the family's convention, and the script is the committed source of truth. +writeFileSync( + join(outDir, "corpus.test.ts"), + `import { readFileSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { lowerMarkdownMath } from "documents.js"; +import { readMarkdownContent } from "markdown-codec"; +import { collectFormulas, formatCorpusReport, runCorpus } from "../../src/harness/corpus"; + +function lowerDocument(markdown: string) { + const { document } = readMarkdownContent(markdown); + return lowerMarkdownMath(document); +} + +describe("compute corpus (generated at-scale worked examples)", () => { + it("measures coverage over every generated document", () => { + const dir = join(import.meta.dirname, "files"); + const files = readdirSync(dir).filter((f) => f.endsWith(".md")); + expect(files.length).toBeGreaterThan(200); + const report = runCorpus( + files.map((file) => ({ + label: file, + document: lowerDocument(readFileSync(join(dir, file), "utf8")), + })), + ); + // The measured report also lands in test/corpus/report.txt -- the measurement's own output artefact, regenerable with the corpus. + writeFileSync( + join(import.meta.dirname, "report.txt"), + formatCorpusReport(report) + "\\n", + ); + process.stdout.write(formatCorpusReport(report) + "\\n"); + // Every generated fixture's stated answer is computed by the generator's own parallel evaluation, so anything short of full coverage names a genuine lowering/evaluation defect. + expect(report.total).toBe(files.length); + expect(report.matched).toBe(report.total); + expect(report.coverage).toBe(1); + }); +}); +`, +); + +console.log( + "compute corpus: " + + documents.length + + " generated worked-example documents under test/corpus/files/", +); diff --git a/packages/document-compute.js/vitest.config.ts b/packages/document-compute.js/vitest.config.ts new file mode 100644 index 000000000..04920c5e5 --- /dev/null +++ b/packages/document-compute.js/vitest.config.ts @@ -0,0 +1,18 @@ +import { defineConfig } from "vitest/config"; + +// Two named projects in one config, filtered by --project in package.json's scripts: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "corpus" (test/corpus/**/*.test.ts) for the optional, gitignored at-scale worked-example layer generated into test/corpus/ by scripts/generate-corpus.mjs, run only by pnpm test:corpus and never part of pnpm test. Before this file existed the package ran configless vitest (whose default include sweeps every directory), which is exactly why the split is explicit now: the corpus layer must never leak into pnpm test the way a default include would let it. +export default defineConfig({ + test: { + // Vitest resolves coverage once for the whole run from this root config, not per project, so it cannot live inside the 'unit' project's own test block; pnpm test:coverage scopes what actually gets measured by filtering to --project unit, which never imports the corpus suite. + coverage: { + provider: "v8", + include: ["src/**/*.ts"], + exclude: ["src/**/*.test.ts"], + reporter: ["text", "html", "cobertura"], + }, + projects: [ + { test: { name: "unit", include: ["src/**/*.test.ts"] } }, + { test: { name: "corpus", include: ["test/corpus/**/*.test.ts"] } }, + ], + }, +}); diff --git a/packages/documents.js/src/latex/lower.test.ts b/packages/documents.js/src/latex/lower.test.ts index af77422ec..774b25f3f 100644 --- a/packages/documents.js/src/latex/lower.test.ts +++ b/packages/documents.js/src/latex/lower.test.ts @@ -176,6 +176,37 @@ describe("lowerLatex mechanical rules", () => { ], }, }, + { + // A minus after a relation signs the FOLLOWING operand, the same unary reading as a leading minus -- found by the generated worked-example corpus, where every negative stated answer degraded its whole equality under the leading-only spelling. + latex: "T = -0.36", + expected: { + kind: "app", + operator: "math:eq", + args: [ + { kind: "sym", id: "symbols:T" }, + { + kind: "app", + operator: "math:negate", + args: [{ kind: "num", numerator: "9", denominator: "25" }], + }, + ], + }, + }, + { + latex: "a + -b", + expected: { + kind: "app", + operator: "math:add", + args: [ + { kind: "sym", id: "symbols:a" }, + { + kind: "app", + operator: "math:negate", + args: [{ kind: "sym", id: "symbols:b" }], + }, + ], + }, + }, { latex: "-x + y", expected: { diff --git a/packages/documents.js/src/latex/lower.ts b/packages/documents.js/src/latex/lower.ts index 81dfd634e..aeb6b697d 100644 --- a/packages/documents.js/src/latex/lower.ts +++ b/packages/documents.js/src/latex/lower.ts @@ -222,29 +222,33 @@ function lowerNodeList( diagnose(context, "latex/operator-unmapped", detail); return unparsed(detail); } - const [firstSegment = [], ...restSegments] = segments; - if (operators.length === 0) { - return lowerTerm(firstSegment, context); - } - const detail = spanOfNodes(context, present); - if (firstSegment.length === 0) { - const leading = operators[0]; - const secondSegment = restSegments[0] ?? []; - if (leading !== SUBTRACT_OPERATOR || secondSegment.length === 0) { - diagnose(context, "latex/operator-placement-unparsed", detail); - return unparsed(detail); - } - return fold( - app(UNARY_MINUS_OPERATOR, [lowerTerm(secondSegment, context)]), - operators.slice(1), - restSegments.slice(1), + const wrapped: FoldSegment[] = segments.map((nodes) => ({ + nodes, + negated: false, + })); + const normalised = normaliseUnaryMinus(operators, wrapped); + const [firstSegment, ...restSegments] = normalised.segments; + if (firstSegment === undefined) { + // Normalisation consumed every segment as unary-minus carriers with nothing left to negate -- an operators-only sequence with no operand at all. + diagnose( context, - detail, + "latex/operator-placement-unparsed", + spanOfNodes(context, present), ); + return unparsed(spanOfNodes(context, present)); + } + if (normalised.operators.length === 0) { + // No operator survived normalisation: a lone segment (possibly negated) is the whole sequence. + return lowerFoldSegment(firstSegment, context); + } + const detail = spanOfNodes(context, present); + if (firstSegment.nodes.length === 0) { + diagnose(context, "latex/operator-placement-unparsed", detail); + return unparsed(detail); } return fold( - lowerTerm(firstSegment, context), - operators, + lowerFoldSegment(firstSegment, context), + normalised.operators, restSegments, context, detail, @@ -279,29 +283,63 @@ const RELATION_OPERATORS: ReadonlySet = new Set( Object.values(RELATION_ATOM_OPERATORS), ); +// One operand segment in fold's input, carrying whether a preceding unary minus makes the segment's folded operand negate: a subtract operator whose FOLLOWING segment is empty (`T = -0.36`, `a + -b`, a leading `-x`) is not a binary subtraction at all -- the minus is the sign of the segment after the empty one. The normalisation pass below rewrites that shape into a subtract-free operator list with the flag set, so fold and foldArithmetic only ever see real binary operators and one flag per segment. +interface FoldSegment { + readonly nodes: readonly TemmlNode[]; + readonly negated: boolean; +} + +// Rewrites every empty-segment-with-a-subtract-after-it into a negation flag on the segment following the subtract (parity-counted, so `a = --b` negates twice), leaving any other empty segment (a genuine placement error, like `a = = b`) for fold's own diagnostic. This generalises the leading-minus-only reading the walk used to special-case: `T = -0.36` degraded the ENTIRE equality under the old spelling, because the empty segment sat after a relation rather than at the head of the sequence -- found by the generated at-scale worked-example corpus (12% of its first run), whose negative stated answers are textbook-ordinary. +function normaliseUnaryMinus( + operators: readonly string[], + segments: readonly FoldSegment[], +): { operators: string[]; segments: FoldSegment[] } { + const outOperators: string[] = []; + const outSegments: FoldSegment[] = []; + let pendingNegate = false; + let skipOperator = false; + for (const [index, segment] of segments.entries()) { + if (index > 0 && !skipOperator) { + const operator = operators[index - 1]; + if (operator !== undefined) { + outOperators.push(operator); + } + } + skipOperator = false; + if (segment.nodes.length === 0 && operators[index] === SUBTRACT_OPERATOR) { + pendingNegate = !pendingNegate; + skipOperator = true; + continue; + } + outSegments.push({ nodes: segment.nodes, negated: pendingNegate }); + pendingNegate = false; + } + return { operators: outOperators, segments: outSegments }; +} + // Standard mathematical convention binds a relation (=, <, \leq, ...) looser than every arithmetic operator, regardless of which side of the relation the arithmetic sits on: `c = a + b` and `a + b = c` both read as eq(add(a,b), c), never add(eq(...), ...) or add(..., eq(...)). A single flat left-to-right fold over the mixed operator list cannot express that -- it folds whichever operator comes first in source order, so `F = m \times a` (relation before arithmetic) folded eq before multiply and produced multiply(eq(F,m), a), a tree with no sound mathematical reading (multiplying an equation by a value). fold instead runs two tiers: foldArithmetic resolves every maximal run of consecutive arithmetic operators into one operand first (unchanged left-to-right arithmetic behaviour within a run), and only then folds those operands together with the relation operators between them, left to right -- so arithmetic always binds first no matter which side of a relation it sits on. function fold( first: MathExpression, operators: readonly string[], - segments: readonly TemmlNode[][], + segments: readonly FoldSegment[], context: LoweringContext, detail: string, ): MathExpression { let runFirst = first; let runOperators: string[] = []; - let runSegments: TemmlNode[][] = []; + let runSegments: FoldSegment[] = []; const operands: MathExpression[] = []; const relations: string[] = []; for (let index = 0; index < operators.length; index += 1) { const operator = operators[index]; - const segmentNodes = segments[index]; - if (operator === undefined || segmentNodes === undefined) { + const segment = segments[index]; + if (operator === undefined || segment === undefined) { throw new Error( "operator and segment lists diverged while folding a lowered sequence", ); } - if (segmentNodes.length === 0) { + if (segment.nodes.length === 0) { diagnose(context, "latex/operator-placement-unparsed", detail); return unparsed(detail); } @@ -310,13 +348,13 @@ function fold( foldArithmetic(runFirst, runOperators, runSegments, context), ); relations.push(operator); - runFirst = lowerTerm(segmentNodes, context); + runFirst = lowerFoldSegment(segment, context); runOperators = []; runSegments = []; continue; } runOperators.push(operator); - runSegments.push(segmentNodes); + runSegments.push(segment); } operands.push(foldArithmetic(runFirst, runOperators, runSegments, context)); @@ -341,23 +379,32 @@ function fold( function foldArithmetic( first: MathExpression, operators: readonly string[], - segments: readonly TemmlNode[][], + segments: readonly FoldSegment[], context: LoweringContext, ): MathExpression { let folded = first; for (let index = 0; index < operators.length; index += 1) { const operator = operators[index]; - const segmentNodes = segments[index]; - if (operator === undefined || segmentNodes === undefined) { + const segment = segments[index]; + if (operator === undefined || segment === undefined) { throw new Error( "operator and segment lists diverged while folding an arithmetic run", ); } - folded = app(operator, [folded, lowerTerm(segmentNodes, context)]); + folded = app(operator, [folded, lowerFoldSegment(segment, context)]); } return folded; } +// One segment's folded operand, with the unary-minus flag normalisation attached (negate wraps the folded term, never the raw nodes -- negation is an operation on the lowered value). +function lowerFoldSegment( + segment: FoldSegment, + context: LoweringContext, +): MathExpression { + const folded = lowerTerm(segment.nodes, context); + return segment.negated ? app(UNARY_MINUS_OPERATOR, [folded]) : folded; +} + // A run of nodes with no binary/relation operator inside: binders and named functions consume the rest of the run, digit runs fold into one numeric literal, and ANY remaining adjacency degrades to one `unparsed` node -- the juxtaposition rule. Juxtaposition is where the issue draws the line between mechanical and context-starved: `mc^2`, `f(x)`, `2(x+1)` all have multiplication AND function application as defensible readings, and LaTeX notation cannot say which, so the run stays visible data with a diagnostic instead of becoming a guess. function lowerTerm( nodes: readonly TemmlNode[], diff --git a/packages/ooxml.js/.gitignore b/packages/ooxml.js/.gitignore index 5903e55d5..67ccec38c 100644 --- a/packages/ooxml.js/.gitignore +++ b/packages/ooxml.js/.gitignore @@ -5,3 +5,4 @@ dist/ coverage/ .eslintcache .turbo/ +test/corpus/ diff --git a/packages/ooxml.js/package.json b/packages/ooxml.js/package.json index 39a4e6a79..b18493d3d 100644 --- a/packages/ooxml.js/package.json +++ b/packages/ooxml.js/package.json @@ -67,7 +67,9 @@ "test:workers": "turbo run _test:workers", "_test:workers": "vitest run --config vitest.workers.config.ts", "prepare": "husky", - "release": "semantic-release" + "release": "semantic-release", + "test:corpus": "turbo run _test:corpus", + "_test:corpus": "vitest run --project corpus" }, "keywords": [ "ooxml", diff --git a/packages/ooxml.js/scripts/generate-xlsx-drawing-corpus.mjs b/packages/ooxml.js/scripts/generate-xlsx-drawing-corpus.mjs new file mode 100644 index 000000000..3da6548a2 --- /dev/null +++ b/packages/ooxml.js/scripts/generate-xlsx-drawing-corpus.mjs @@ -0,0 +1,197 @@ +#!/usr/bin/env node +// Generates this package's gitignored real-producer xlsx-drawing corpus under test/corpus/, ready for `pnpm test:corpus` -- the corpus gate src/typed/xlsx/drawings.ts itself states ("Real-producer verification is outstanding"). The producer is LibreOffice's own Calc Office Open XML export: each fixture is a flat-ODS spreadsheet authored with a cell-embedded image (declared column widths and row heights, varied anchor cells and frame sizes), converted headlessly through `soffice --convert-to xlsx`, so the drawing part's anchor markup and the worksheet grid it resolves against are both genuine application output rather than hand-built ECMA-376. Run from the package root: `node scripts/generate-xlsx-drawing-corpus.mjs` (soffice at /opt/homebrew/bin/soffice or SOFFICE in the environment; overwrites test/corpus/ wholesale). +// +// One producer boundary this corpus deliberately does not cover: Calc never emits an xdr:absoluteAnchor (its export normalises every drawing to twoCellAnchor spellings -- verified empirically against both cell-embedded and sheet-level frames), so the absoluteAnchor re-basing path keeps its hand-built unit fixtures as its only verification; that gap is stated in the package README rather than papered over. +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const packageRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); +const outDir = join(packageRoot, "test", "corpus"); +const soffice = process.env.SOFFICE ?? "/opt/homebrew/bin/soffice"; + +// A 2x2 solid-red PNG (CRC-correct; LibreOffice's libpng silently drops a bad-IHDR-CRC picture). +const TINY_PNG = + "iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAIAAAD91JpzAAAAEElEQVR4nGP4z8AARAwQCgAf7gP9i18U1AAAAABJRU5ErkJggg=="; + +const NS = + 'xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" ' + + 'xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0" ' + + 'xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" ' + + 'xmlns:draw="urn:oasis:names:tc:opendocument:xmlns:drawing:1.0" ' + + 'xmlns:svg="urn:oasis:names:tc:opendocument:xmlns:svg-compatible:1.0"'; + +// One fixture: the anchor cell (row/column indices), the frame's cm size, and the sheet's own declared column widths (cm) and row heights (cm) the anchor geometry resolves against. +const FIXTURES = [ + { + name: "origin-anchor", + row: 0, + column: 0, + widthCm: 4, + heightCm: 3, + columnWidthsCm: [3, 3, 3], + rowHeightsCm: [1, 1, 1], + expect: { anchorRow: 0, anchorColumn: 0 }, + }, + { + name: "mid-sheet-anchor", + row: 2, + column: 1, + widthCm: 4, + heightCm: 3, + columnWidthsCm: [3, 2.5, 3.5, 3], + rowHeightsCm: [1, 1.25, 0.75, 1], + expect: { anchorRow: 2, anchorColumn: 1 }, + }, + { + name: "wide-frame", + row: 1, + column: 0, + widthCm: 6, + heightCm: 2, + columnWidthsCm: [2, 2, 2, 2, 2], + rowHeightsCm: [1, 1, 1], + expect: { anchorRow: 1, anchorColumn: 0 }, + }, +]; + +function fixtureFods(fixture) { + const columns = fixture.columnWidthsCm + .map( + (cm) => + ``, + ) + .join(""); + // The column/row declared sizes ride on automatic styles, one per distinct width. + const colStyles = [...new Set(fixture.columnWidthsCm)] + .map( + (cm, i) => + ``, + ) + .join(""); + const rowStyles = [...new Set(fixture.rowHeightsCm)] + .map( + (cm) => + ``, + ) + .join(""); + const rows = fixture.rowHeightsCm + .map((cm, r) => { + const cells = fixture.columnWidthsCm + .map((_, c) => { + const label = `r${r}c${c}`; + const image = + r === fixture.row && c === fixture.column + ? `${TINY_PNG}` + : ""; + return `${label}${image}`; + }) + .join(""); + return `${cells}`; + }) + .join(""); + return ( + '\n' + + `\n` + + " " + + colStyles + + rowStyles + + " \n" + + " " + + '' + + columns + + rows + + "" + + " \n" + ); +} + +// The corpus harness: reads each converted .xlsx through this package's own package decode + spreadsheet reader and asserts the image's anchor cell and frame against what the fods source authored. Regenerated by the script alongside the fixtures -- the whole test/corpus/ layer is local-only by the family's convention, and the script is the committed source of truth. +const CORPUS_TEST = `import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { decodePackage, readXlsxContent } from "../../src/index"; + +interface Fixture { + file: string; + expect: { + anchorRow: number; + anchorColumn: number; + widthCm: number; + heightCm: number; + }; +} + +const MANIFEST = JSON.parse( + readFileSync(join(import.meta.dirname, "manifest.json"), "utf8"), +) as readonly Fixture[]; + +// Points per centimetre, matching the generator's authoring unit; the tolerance absorbs Calc's cm-to-EMU rounding. +const PT_PER_CM = 72 / 2.54; + +describe("xlsx drawing corpus (LibreOffice-produced Calc output)", () => { + for (const { file, expect: e } of MANIFEST) { + it(file + " resolves its picture anchor and frame against the producer grid", () => { + const bytes = new Uint8Array(readFileSync(join(import.meta.dirname, file))); + const content = readXlsxContent(decodePackage(bytes)); + if (content.kind !== "spreadsheet") { + throw new Error("expected a spreadsheet document"); + } + const images = content.sheets[0]!.images; + expect(images, "sheet pictures recovered").toHaveLength(1); + const image = images[0]!; + expect(image.anchorRow).toBe(e.anchorRow); + expect(image.anchorColumn).toBe(e.anchorColumn); + // The producer's own a:ext in the drawing part states the frame in EMU; the reader's grid-derived frame must agree with the authored cm within the cm-to-EMU rounding (1pt tolerance covers 32k EMU of drift). + expect(image.widthPt).toBeCloseTo(e.widthCm * PT_PER_CM, 0); + expect(image.heightPt).toBeCloseTo(e.heightCm * PT_PER_CM, 0); + }); + } +}); +`; + +rmSync(outDir, { recursive: true, force: true }); +mkdirSync(outDir, { recursive: true }); +const staging = join(outDir, ".staging"); +mkdirSync(staging, { recursive: true }); + +const manifest = []; +for (const fixture of FIXTURES) { + const src = join(staging, fixture.name + ".fods"); + writeFileSync(src, fixtureFods(fixture)); + execFileSync( + soffice, + [ + "--headless", + "--norestore", + "--convert-to", + "xlsx", + "--outdir", + staging, + src, + ], + { stdio: "pipe" }, + ); + rmSync(src, { force: true }); + writeFileSync( + join(outDir, fixture.name + ".xlsx"), + readFileSync(join(staging, fixture.name + ".xlsx")), + ); + manifest.push({ + file: fixture.name + ".xlsx", + expect: { + anchorRow: fixture.expect.anchorRow, + anchorColumn: fixture.expect.anchorColumn, + widthCm: fixture.widthCm, + heightCm: fixture.heightCm, + }, + }); +} +rmSync(staging, { recursive: true, force: true }); + +writeFileSync(join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2)); +writeFileSync(join(outDir, "corpus.test.ts"), CORPUS_TEST); +console.log( + "xlsx drawing corpus: " + manifest.length + " fixtures under test/corpus/", +); diff --git a/packages/ooxml.js/src/typed/xlsx/drawings.ts b/packages/ooxml.js/src/typed/xlsx/drawings.ts index fcff80aad..24255a0f7 100644 --- a/packages/ooxml.js/src/typed/xlsx/drawings.ts +++ b/packages/ooxml.js/src/typed/xlsx/drawings.ts @@ -28,7 +28,7 @@ import { // A worksheet's drawing layer (xl/drawings/drawingN.xml, reached through the worksheet's own relationships): the xlsx counterpart of pptx's chart/SmartArt/OLE readers. A chart graphic frame's cached series/category model is read through the SAME chart reader the pptx side uses (readChartTable), and lands as a ContentEmbeddedObject with objectKind 'chart' -- the one member that names what the frame held rather than a ContentDocument kind, carrying the cached model as a small spreadsheet document (one sheet whose cells are that table), because a sheet is the honest document-granularity spelling of tabular data and a xlsx sheet has no block flow to host a table block the way a pptx shape does. A picture (xdr:pic) resolves its a:blip through the drawing part's own relationships to the sniffed media bytes and lands as a ContentSheetImage -- the same blip-resolution contract as the pptx picture reader, anchor fields and frame resolved through the same grid geometry the chart row uses. // -// Scope: all three anchor spellings a drawing part carries (charts and pictures), every spelling resolving to one placement shape -- a from-marker positions a two-cell or one-cell anchor through the same grid geometry, with the frame's size the to-marker difference (two-cell) or the anchor's own xdr:ext (one-cell, Excel's "Move, but don't size with cells" spelling for inserted pictures); an absoluteAnchor's page-absolute xdr:pos is re-based into the cell-relative anchor vocabulary through that same geometry's inverse (the nearest-cell landing #776 decides on, rather than a schema extension -- the geometry is a bijection between cell-plus-offset and absolute position, and the frame keeps the absolute position verbatim, so the re-basing loses nothing). Real-producer verification is outstanding: the fixtures this is built against are hand-built ECMA-376 markup, the corpus gate the construct inventory itself states. +// Scope: all three anchor spellings a drawing part carries (charts and pictures), every spelling resolving to one placement shape -- a from-marker positions a two-cell or one-cell anchor through the same grid geometry, with the frame's size the producer's own transform extent when the anchor child states one (real producers' exact-EMU authority -- the grid's to-marker difference is the fallback, and the two genuinely disagree wherever declared column widths ride xlsx's approximate character units) (two-cell) or the anchor's own xdr:ext (one-cell, Excel's "Move, but don't size with cells" spelling for inserted pictures); an absoluteAnchor's page-absolute xdr:pos is re-based into the cell-relative anchor vocabulary through that same geometry's inverse (the nearest-cell landing #776 decides on, rather than a schema extension -- the geometry is a bijection between cell-plus-offset and absolute position, and the frame keeps the absolute position verbatim, so the re-basing loses nothing). Real-producer verification exists for the anchor spellings a real producer emits: pnpm test:corpus runs the gitignored LibreOffice-produced Calc corpus (scripts/generate-xlsx-drawing-corpus.mjs), whose genuine twoCellAnchor editAs="oneCell" output verified the anchor resolution and exposed that a oneCell-anchored picture's frame must come from the producer's own transform extent (the to-marker difference disagrees by the character-unit column-width approximation underneath) -- the readChildTransformExtEmu/editAs rule below. What stays hand-built-only: xdr:absoluteAnchor, because Calc's export normalises every drawing to twoCellAnchor spellings and no accessible producer here emits one. const CHART_GRAPHIC_URI = "http://schemas.openxmlformats.org/drawingml/2006/chart"; @@ -218,6 +218,31 @@ function readAnchorExtEmu( return { cxEmu: numericAttr(ext, "cx"), cyEmu: numericAttr(ext, "cy") }; } +// The anchor child's own DrawingML transform extent (xdr:pic/xdr:sp/xdr:graphicFrame > xdr:spPr > a:xfrm > a:ext, cx/cy EMU) -- the frame size the producer itself states, in exact EMU, rather than the to-marker difference the grid geometry derives. Real producers write both (Calc and Word alike), and the two disagree wherever the worksheet's declared column widths are stated in xlsx's approximate character units: the producer's own EMU is the authority for the frame's size, exactly as the one-cell and absolute spellings already treat their xdr:ext. +function readChildTransformExtEmu( + anchor: XmlElement, +): { readonly cxEmu: number; readonly cyEmu: number } | undefined { + for (const child of anchor.children) { + if (child.type !== "element") { + continue; + } + const spPr = childrenWithTag(child, "xdr:spPr")[0]; + if (spPr === undefined) { + continue; + } + const xfrm = childrenWithTag(spPr, "a:xfrm")[0]; + if (xfrm === undefined) { + continue; + } + const ext = childrenWithTag(xfrm, "a:ext")[0]; + if (ext === undefined) { + continue; + } + return { cxEmu: numericAttr(ext, "cx"), cyEmu: numericAttr(ext, "cy") }; + } + return undefined; +} + // One anchor's placement: a two-cell anchor is positioned by its from-marker and sized by the to-marker difference, a one-cell anchor by its from-marker and its own xdr:ext, an absolute anchor by its page-absolute xdr:pos (re-based into the cell anchor vocabulary through the grid geometry's own inverse, since ContentSheetImage and ContentEmbeddedObject anchor cell-relatively) and its own xdr:ext. Undefined when the anchor's own geometry is malformed (a missing marker, pos, or ext), which skips the anchor the way the walk always has. function readAnchorPlacement( anchor: XmlElement, @@ -231,11 +256,21 @@ function readAnchorPlacement( } const xPt = geometry.xPt(from.column, from.colOffEmu); const yPt = geometry.yPt(from.row, from.rowOffEmu); + // editAs governs which size statement is the semantic one: "oneCell" means move-but-not-size-with-cells, so the shape's own transform extent is the frame (the to-marker is Calc's spelling habit for it and disagrees with the character-unit column widths underneath -- verified against real producer output); "twoCell" (also ECMA's default) means the frame IS the to-marker difference, resizing with the grid, so the grid rules; "absolute" sizes independently of both. + const editAs = attr(anchor, "editAs") ?? "twoCell"; + const childExt = + editAs === "oneCell" ? readChildTransformExtEmu(anchor) : undefined; return { xPt, yPt, - widthPt: geometry.xPt(to.column, to.colOffEmu) - xPt, - heightPt: geometry.yPt(to.row, to.rowOffEmu) - yPt, + widthPt: + childExt !== undefined + ? emuToPt(childExt.cxEmu) + : geometry.xPt(to.column, to.colOffEmu) - xPt, + heightPt: + childExt !== undefined + ? emuToPt(childExt.cyEmu) + : geometry.yPt(to.row, to.rowOffEmu) - yPt, anchorRow: from.row, anchorColumn: from.column, offsetXPt: emuToPt(from.colOffEmu), diff --git a/packages/ooxml.js/vitest.config.ts b/packages/ooxml.js/vitest.config.ts index 623e5767c..2924a02f9 100644 --- a/packages/ooxml.js/vitest.config.ts +++ b/packages/ooxml.js/vitest.config.ts @@ -1,6 +1,6 @@ import { defineConfig } from "vitest/config"; -// Two named projects in one config, filtered by --project in package.json's scripts rather than a separate config file: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. +// Two named projects in one config, filtered by --project in package.json's scripts rather than a separate config file: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. A "corpus" project (test/corpus/**/*.test.ts) holds the optional, gitignored real-producer layer -- LibreOffice-produced Calc xlsx drawings generated into test/corpus/ by scripts/generate-xlsx-drawing-corpus.mjs, run only by pnpm test:corpus. export default defineConfig({ test: { coverage: { @@ -12,6 +12,7 @@ export default defineConfig({ projects: [ { test: { name: "unit", include: ["src/**/*.test.ts"] } }, { test: { name: "smoke", include: ["test/smoke.test.mjs"] } }, + { test: { name: "corpus", include: ["test/corpus/**/*.test.ts"] } }, ], }, }); diff --git a/packages/rtf-codec/README.md b/packages/rtf-codec/README.md index f9906249f..29fd4f9f2 100644 --- a/packages/rtf-codec/README.md +++ b/packages/rtf-codec/README.md @@ -4,7 +4,7 @@ > A hand-written, dependency-minimal Rich Text Format codec: reads RTF into the shared [document-schema.js](../document-schema.js/README.md) content pivot, and writes deterministic, 7-bit-ASCII RTF back out. Built against Microsoft's own [RTF Specification, version 1.9.1](#the-specification) and [Zod 4](https://zod.dev), with no third-party RTF library. -**Status: under active development.** The read and write paths described below are implemented and tested, but this package is new and has not yet been exercised against a real-world corpus. [Scope](#scope) states exactly what is handled and what is not; nothing in this README describes work that is planned rather than done. +**Status: under active development.** The read and write paths described below are implemented and tested, including against a real-producer corpus: `pnpm test:corpus` runs the gitignored `test/corpus/` suite against LibreOffice-produced RTF (flat-ODT sources spanning runs, colour, headings, lists, tables, and alignment, converted through Writer's own RTF filter by `scripts/generate-corpus.mjs`) -- the corpus validated the reader cleanly across all eight fixtures with no defects found. [Scope](#scope) states exactly what is handled and what is not; nothing in this README describes work that is planned rather than done. Every construct that remains unhandled is either a gap in `document-schema.js` rather than in this codec (superscript/subscript and text direction have no field to land in), or something RTF itself does not specify at all beyond its own form-field vocabulary (a docx-style rich-text SDT has no RTF spelling of any kind) — see [Deliberately not handled](#deliberately-not-handled), which says which of the two each row is. diff --git a/packages/rtf-codec/package.json b/packages/rtf-codec/package.json index de2b71f98..61562aa4e 100644 --- a/packages/rtf-codec/package.json +++ b/packages/rtf-codec/package.json @@ -67,7 +67,9 @@ "test:smoke": "turbo run _test:smoke", "_test:smoke": "vitest run --project smoke", "prepare": "husky", - "release": "semantic-release" + "release": "semantic-release", + "test:corpus": "turbo run _test:corpus", + "_test:corpus": "vitest run --project corpus" }, "keywords": [ "rtf", diff --git a/packages/rtf-codec/scripts/generate-corpus.mjs b/packages/rtf-codec/scripts/generate-corpus.mjs new file mode 100644 index 000000000..5bd550678 --- /dev/null +++ b/packages/rtf-codec/scripts/generate-corpus.mjs @@ -0,0 +1,278 @@ +#!/usr/bin/env node +// Generates this package's gitignored real-producer .rtf corpus under test/corpus/, ready for `pnpm test:corpus` (the package shipped no corpus layer at all before this). The producer is LibreOffice's own RTF export filter: each fixture is authored as a flat-ODT source and converted headlessly through `soffice --convert-to rtf`, so every corpus file is genuine application output rather than hand-authored control words -- exactly the "never been exercised against real application output" gap this package's own README names. Run from the package root: `node scripts/generate-corpus.mjs` (soffice at /opt/homebrew/bin/soffice or SOFFICE in the environment; overwrites test/corpus/ wholesale). The regenerated corpus.test.ts carries each fixture's expectations, authored beside the fixture's construction here. +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const packageRoot = join(dirname(fileURLToPath(import.meta.url)), ".."); +const outDir = join(packageRoot, "test", "corpus"); +const soffice = process.env.SOFFICE ?? "/opt/homebrew/bin/soffice"; + +const NS = + 'xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" ' + + 'xmlns:style="urn:oasis:names:tc:opendocument:xmlns:style:1.0" ' + + 'xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0" ' + + 'xmlns:table="urn:oasis:names:tc:opendocument:xmlns:table:1.0" ' + + 'xmlns:fo="urn:oasis:names:tc:opendocument:xmlns:xsl-fo-compatible:1.0" ' + + 'xmlns:svg="urn:oasis:names:tc:opendocument:xmlns:svg-compatible:1.0" ' + + 'xmlns:xlink="http://www.w3.org/1999/xlink"'; + +const FIXTURES = [ + { + name: "plain", + body: "Corpus plain paragraph.", + expect: { text: ["Corpus plain paragraph."] }, + }, + { + name: "runs", + body: + "" + + 'bold run and ' + + 'italic run and ' + + 'underlined run' + + "", + expect: { + runs: [ + { text: "bold run", bold: true }, + { text: "italic run", italic: true }, + { text: "underlined run", underline: true }, + ], + }, + }, + { + name: "colour", + body: 'red text', + expect: { runs: [{ text: "red text", color: { r: 1, g: 0, b: 0 } }] }, + }, + { + name: "headings", + body: + 'Chapter One' + + 'Section A' + + "Body under the section.", + expect: { + headings: [ + { text: "Chapter One", level: 1 }, + { text: "Section A", level: 2 }, + ], + }, + }, + { + name: "lists", + body: + 'bullet onebullet two' + + 'step onestep two', + expect: { + listItems: ["bullet one", "bullet two", "step one", "step two"], + }, + }, + { + name: "table", + body: + '' + + "" + + "" + + "alpha" + + "beta" + + "" + + "" + + "gamma" + + "delta" + + "" + + "", + expect: { + tables: [{ rows: 2, cells: ["alpha", "beta", "gamma", "delta"] }], + }, + }, + { + name: "multiple-paragraphs", + body: + "First paragraph." + + "Second paragraph." + + "Third paragraph.", + expect: { paragraphs: 3, text: ["First paragraph.", "Third paragraph."] }, + }, + { + name: "alignment", + body: 'Centred text.', + expect: { text: ["Centred text."], centered: "Centred text." }, + }, +]; + +const AUTOSTYLES = + " " + + '' + + '' + + '' + + '' + + '' + + '' + + '' + + " "; + +function fodt(body) { + return ( + '\n\n' + + AUTOSTYLES + + "\n " + + body + + "\n" + ); +} + +// The corpus harness: reads each converted .rtf through this package's own reader and asserts the manifest's expectations against the recovered ContentDocument. Regenerated by the script alongside the fixtures -- the whole test/corpus/ layer is local-only by the family's convention, and the script is the committed source of truth. +const CORPUS_TEST = `import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { readRtfContent } from "../../src/index"; + +const MANIFEST = JSON.parse( + readFileSync(join(import.meta.dirname, "manifest.json"), "utf8"), +) as readonly { + file: string; + expect: Record; +}[]; + +describe("rtf corpus (LibreOffice-produced RTF)", () => { + for (const { file, expect: e } of MANIFEST) { + it(file + " reads back what its flat-ODT source authored", () => { + const text = new TextDecoder().decode( + new Uint8Array(readFileSync(join(import.meta.dirname, file))), + ); + const { document } = readRtfContent(text); + if (document.kind !== "wordprocessing") { + throw new Error("expected a wordprocessing document"); + } + const blocks = document.sections[0]!.blocks; + const paragraphs = blocks.filter( + (b): b is Extract => + b.kind === "paragraph", + ); + const wholeText = paragraphs + .flatMap((p) => p.runs.map((r) => r.text)) + .join(" ") + .replace(/\\s+/g, " ") + .trim(); + if (Array.isArray(e.text)) { + for (const needle of e.text as string[]) { + expect(wholeText).toContain(needle); + } + } + if (typeof e.paragraphs === "number") { + expect(paragraphs).toHaveLength(e.paragraphs); + } + if (Array.isArray(e.runs)) { + const all = paragraphs.flatMap((p) => p.runs); + for (const want of e.runs as { + text: string; + bold?: boolean; + italic?: boolean; + underline?: boolean; + color?: { r: number; g: number; b: number }; + }[]) { + const run = all.find((r) => r.text === want.text); + expect(run, "run " + want.text + " recovered").toBeDefined(); + if (want.bold !== undefined) expect(run!.bold ?? false).toBe(want.bold); + if (want.italic !== undefined) + expect(run!.italic ?? false).toBe(want.italic); + if (want.underline !== undefined) + expect(run!.underline ?? false).toBe(want.underline); + if (want.color !== undefined) { + expect(run!.color, "colour on " + want.text).toBeDefined(); + expect(run!.color!.r).toBeCloseTo(want.color.r, 1); + expect(run!.color!.g).toBeCloseTo(want.color.g, 1); + expect(run!.color!.b).toBeCloseTo(want.color.b, 1); + } + } + } + if (Array.isArray(e.headings)) { + for (const want of e.headings as { text: string; level: number }[]) { + const heading = paragraphs.find( + (p) => p.runs.map((r) => r.text).join("") === want.text, + ); + expect(heading, "heading " + want.text).toBeDefined(); + expect(heading!.headingLevel).toBe(want.level); + } + } + if (Array.isArray(e.listItems)) { + for (const item of e.listItems as string[]) { + const para = paragraphs.find( + (p) => p.runs.map((r) => r.text).join("") === item, + ); + expect(para, "list item " + item).toBeDefined(); + expect(para!.list, "membership on " + item).toBeDefined(); + } + } + if (Array.isArray(e.tables)) { + const tables = blocks.filter( + (b): b is Extract => b.kind === "table", + ); + expect(tables).toHaveLength((e.tables as unknown[]).length); + for (const [ti, want] of ( + e.tables as { rows: number; cells: string[] }[] + ).entries()) { + const table = tables[ti]!; + expect(table.rows).toHaveLength(want.rows); + const cells = table.rows.flatMap((r) => r.cells); + expect( + cells.map((c) => + c.blocks + .map((b) => + b.kind === "paragraph" + ? b.runs.map((r) => r.text).join("") + : "", + ) + .join(""), + ), + ).toEqual(want.cells); + } + } + if (typeof e.centered === "string") { + const para = paragraphs.find( + (p) => p.runs.map((r) => r.text).join("") === e.centered, + ); + expect(para, "centred paragraph " + e.centered).toBeDefined(); + expect(para!.alignment).toBe("center"); + } + }); + } +}); +`; + +rmSync(outDir, { recursive: true, force: true }); +mkdirSync(outDir, { recursive: true }); +const staging = join(outDir, ".staging"); +mkdirSync(staging, { recursive: true }); + +const manifest = []; +for (const fixture of FIXTURES) { + const src = join(staging, fixture.name + ".fodt"); + writeFileSync(src, fodt(fixture.body)); + execFileSync( + soffice, + [ + "--headless", + "--norestore", + "--convert-to", + "rtf", + "--outdir", + staging, + src, + ], + { stdio: "pipe" }, + ); + rmSync(src, { force: true }); + writeFileSync( + join(outDir, fixture.name + ".rtf"), + readFileSync(join(staging, fixture.name + ".rtf")), + ); + manifest.push({ file: fixture.name + ".rtf", expect: fixture.expect }); +} +rmSync(staging, { recursive: true, force: true }); + +writeFileSync(join(outDir, "manifest.json"), JSON.stringify(manifest, null, 2)); +writeFileSync(join(outDir, "corpus.test.ts"), CORPUS_TEST); +console.log("rtf corpus: " + manifest.length + " fixtures under test/corpus/"); diff --git a/packages/rtf-codec/vitest.config.ts b/packages/rtf-codec/vitest.config.ts index 573b8234a..7a3f4ecfa 100644 --- a/packages/rtf-codec/vitest.config.ts +++ b/packages/rtf-codec/vitest.config.ts @@ -1,6 +1,6 @@ import { defineConfig } from "vitest/config"; -// Two named projects in one config, filtered by --project in package.json's scripts: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. There is deliberately no "corpus" project yet, unlike markdown-codec's and pdf-codec's: this package has no gitignored real-world RTF conformance corpus to point one at, and declaring an empty project would report a passing suite that checks nothing. +// Two named projects in one config, filtered by --project in package.json's scripts: "unit" (src/**/*.test.ts) for pnpm test/test:watch, and "smoke" (test/smoke.test.mjs, which imports from dist/) only ever run by pnpm test:smoke, right after tsdown rebuilds dist/. A "corpus" project (test/corpus/**/*.test.ts) holds the optional, gitignored real-producer conformance layer -- LibreOffice-produced RTF documents generated into test/corpus/ by scripts/generate-corpus.mjs, run only by pnpm test:corpus and never part of pnpm test. export default defineConfig({ test: { // Vitest resolves coverage once for the whole run from this root config, not per project, so it cannot live inside the 'unit' project's own test block; pnpm test:coverage scopes what actually gets measured by filtering to --project unit, which never imports the smoke suite. @@ -13,6 +13,7 @@ export default defineConfig({ projects: [ { test: { name: "unit", include: ["src/**/*.test.ts"] } }, { test: { name: "smoke", include: ["test/smoke.test.mjs"] } }, + { test: { name: "corpus", include: ["test/corpus/**/*.test.ts"] } }, ], }, });