The page reader repeats a line into every column of a ruled grid it crosses

[lemarier]

The page reader (layout-v1, #216) writes a line of prose into every column of a ruled grid it crosses, and interleaves two overlapping lines letter by letter. On three of wave 2's documents most of the conversion is this repetition, and the text reader pays for every window of it.

Evidence

Energizer's Homepower installation manual converts to 931,504 characters where toMarkdown gave 65,746: 13 windows became 172. 80% of the new text is in lines over 2,000 characters. The longest is 23,646 characters and 106 cells, each holding the same sentence:

| The installer shall identify a suitable location for the Energizer Homepower HP-6 Series system, maintain | The installer shall identify a suitable location for the Energizer Homepower HP-6 Series system, maintain | …

and overlapping lines come out interleaved:

| 43HPePr-f6oSrm uannitc ise cdoen-rsaitdinegre edx ap escutbe-dsy isnt eexmt roefm the ea Mmbaiien nUt nteitm …

Document	Windows before	After	In lines over 2,000 characters
Energizer Homepower installation manual	13	172	80%
Magnum MPXD extension kit manual	21	204	71%
EG4 6000XP system wiring diagrams	10	72	63%

The other 55 documents wave 2 read past sixty windows are under 50%. Across 120 sampled documents the converter costs ×1.35 in windows, so this is confined, but on these three it is most of what was read.

Likely cause

A cell whose text crosses one of the page's ruled lines is taken to belong to both columns, which is what a merged cell is (rulesOn and gridOf in apps/worker/src/layout.ts). On a page whose rules are a drawing, a wiring diagram or a form rather than a table, every line of prose crosses them all and is repeated into each. The interleaving looks like two runs of text at the same angle and height grouped into one line by vertical overlap.

Done when

A page whose rules do not form a table is written as prose, and a line is never repeated into more columns than a table on that page has. These three documents convert to roughly the size of their text.

[lemarier]

#237 fixes the repeated lines: rules now keep their height and divide only their own table's rows, and letter strokes, box sides and frames are no longer taken for rules. On 46 PDFs, lines with a cell written three times or more went from 4,875 to 255 and the text shrank to 0.63 of its size, with no word lost.

Two lines shuffled into one are still open here. What was tried, and why each was dropped:

• Cause. rowsOf puts a character in the first row whose box it overlaps by 40%, and the box grows with every character taken. Brackets and a cell wrapped over several lines in another column grow it until it reaches the next line, and characters of both are sorted by left into one row. The Energizer Solar EHP owner's manual, page 16, reads Phosphate (P1o-)l,y Hetehxyaleflnueo ro- for Phosphate (1-), Hexafluoro- under Polyethylene, 18 points above it. NOCO's AX65 guide has whole paragraphs shuffled this way (page 28 German, page 426 Chinese), and Victron's off-grid booklet, page 97, hides its default charge voltages (14,4 / 28,8 / 43,2 / 57,6V) in it.
• A band per row (a character joins only if its middle is within the row's average top and bottom): fixes those, but loses the letters that stand out of the band when they arrive before their own line has a row. Characters are taken tallest first, so the two ls of Victron's Nominal Battery voltage (page 57) went to the row above and the line read Nomina Battery vo tage; a French orange lost its g; Arabic lost its diacritics. Victron's spaces are boxes 0.0064 points tall and need the old rule, or words run together (Maxoperational, page 53).
• Refusing a character that would stand over a letter of like size on another line, with rows otherwise built as before: too late. The first letters of the lower line join before any of them lands on one of the upper line, so the words are split instead of shuffled.
• Likely way through: group characters by baseline, not by box. FPDFText_GetCharOrigin gives each character's origin and is in the PDFium build the Worker ships. Characters of one line share a baseline whatever their box; a superscript sits about a third of an em off it and the next line a whole line away. What needs care: text turned a quarter circle (turned), and a value wrapped onto two lines beside a one-line label, which box overlap keeps with its label today (NOCO's Accutypes, AX65 guide page 96, Dutch).

Measure against main with a word-by-word check of every changed page, as #237 did: the band version passed its tests and the fixtures above, and only the word check caught the letters it lost.