Learned layout classifier - #6
Open
ivanvanderbyl wants to merge 53 commits into
Open
Conversation
…uards Scientific papers set in the base-14 fonts extracted as one collapsed word per line: the PDF spec lets a document omit /Widths for the standard fonts because a conforming reader is required to know their metrics, and the face-less port had none, so every glyph advanced by zero. - Add the Adobe Core 14 AFM advance widths and vertical metrics, generated from the PDFBox AFM files and cross-checked glyph-for-glyph against pdf.js. They only fill metrics the document left unset, so an explicit /Widths, /MissingWidth, or an embedded program still wins. - Scale the Type 3 /FontBBox by the FontMatrix into glyph space, so Type 3 glyph boxes stop collapsing to a baseline sliver under a y-flipping matrix. - Emit a chord segment for the Bezier path operators. A figure drawn only from curves previously reported no geometry at all. - Claim characters exclusively when re-extracting merged cells, so a glyph whose box straddles two regions is emitted once rather than twice. - Reject running-header, list-item, equation-debris, italic-dominant and sentence-shaped lines as headings; identify figure regions by shape rather than page-area fraction. - Add pdf-inspector to the benchmarked tool set. DPBench (200 PDFs): extraction 0.9209 -> 0.9216, reading order 0.8933 -> 0.8938, heading level 0.7681 -> 0.7708, TEDS unchanged, 0 errors, no measurable latency change. Also adds five implementation plans for the remaining defect classes; a four-way visual audit of all 55 pages attributes ~60% of remaining defects to TeX math font encodings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rints dvips embeds Computer Modern fonts as Type 3 bitmaps with no /ToUnicode and synthetic glyph names, so their char codes follow the TeX encodings (OT1/OML/OMS/OMX) rather than ASCII: minus signs vanished, Greek letters extracted as control bytes, and omega/arrows extracted as '!'. Identify the encoding from font metrics alone: fit the font's /Widths vector (scale-free least squares) against the published Computer Modern TFM width tables and accept the best encoding only when it has >= 4 glyphs of evidence, <= 2% relative width error, and a >= 4-point margin over every other encoding. Gate the whole mechanism on the absence of /ToUnicode and on glyph names that carry no information beyond the char code (#XX hex form, identity single chars, dvips's comma at code 0). When ambiguous, no encoding is chosen and behaviour is unchanged. The resolved table is applied only as a fallback after /ToUnicode and the existing encoding path, never overriding a successful mapping. On the 55-page dvips-produced entropy.pdf: minus signs 0 -> 313, "<=" 0 -> 47, Greek letters restored (alpha 63, pi 64, ...), invisible control bytes 585 -> 1, prose byte-identical. DPBench (200 PDFs): all accuracy metrics and outputs byte-identical, 0 errors - no false positives corpus-wide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ines
Display equations and inline stacked constructs destroyed surrounding prose
in two coupled ways, both confirmed by cell dumps of entropy.pdf p22/p37:
1. Glyph theft: big delimiters (parens, integrals) carry rects 2-3x taller
than their line, reaching through neighbouring prose lines. Sequential
first-query-wins claiming re-extraction let such a rect claim prose
glyphs it merely grazed ('possible information sources' became
'possiblnformation surces'; 'variables' was shredded to 'bles').
Replaced with a batched best-overlap partition: every character goes to
the query box covering the greatest fraction of its glyph box (ties:
horizontal fraction, vertical-centre proximity, box order), so ownership
is order-independent and a glyph stays with its own line's rect.
New: text.GetTextByRectsExclusive, pdf.MergeFragmentedCellsExclusive.
2. Line bridging: a tall cell inflated the line-clustering band until the
overlap fallback absorbed the next prose line, interleaving equation
glyphs with prose left-to-right. Bridged lines are now re-split by their
dominant-height cells (median-based outlier detection); tall outliers
attach to the baseline cluster they overlap best, top-anchored on ties.
Also inlines fraction-style stacks in reading order: a small fragment line
whose cells fit inside an interior gap of a vertically-interpenetrating
neighbour folds in at its x-position, with column mates reading
top-to-bottom (numerator before denominator, never hoisted to line start).
All heuristics are geometric (box heights, overlaps, gaps) with no
character-content signals. entropy.pdf: all seven audited corruptions gone,
their correct words restored; token stats stable (28411 -> 28335, avg len
4.357 -> 4.402).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation Three document-general fixes for display mathematics being gridded into Markdown tables and real table rows being emitted two or three times: - render: emit a spanned cell's text once at its anchor slot instead of in every covered row/column (Markdown has no rowspan), and drop fully-empty body rows. This removes the duplicated rows in ruled and grid-built tables (Table I on page 40 of entropy.pdf was emitted 2-3x per row). - table: add a gutter-persistence gate for line-built borderless tables. A real table's column gutters are whitespace corridors that persist down the whole table; prose flowing across the claimed boundaries (corridor-midpoint coverage) or columns whose robust content extents interpenetrate (degenerate corridors) mark the grid as imaginary. Tables without line-level column evidence are re-judged on word tokens. Both positive (suppression) and negative (genuine table kept) cases are covered by tests. - pdf: extract standalone marginal page-number cells before table detection so tables and equation regions at the page edge cannot swallow the page number into a cell. entropy.pdf: 19 table blocks / 121 rows -> 8 blocks / 44 rows; all four real tables survive with each row exactly once; released equations flow as text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2% error bound was calibrated against whole non-TeX fonts, which are rejected by a wide margin. It did not hold for the small subsets real PDFs commonly embed: sampling random k-glyph subsets of the Adobe base-14 metrics, Times-Roman was misidentified 8.4% of the time at k=4. Raise the minimum evidence to 5 glyphs and tighten the error bound to 1%, which is still above the <= 0.9% at which true matches calibrate. Measured Times-Roman subset false-identification falls from 2.16% to 0.06%, and every real dvips subset under test is still accepted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review of the display-equation change found one blocker and two correctness issues; all fixed, plus dead-path removal. Blocker: foldStackedFragmentLines could merge two table rows and interleave their columns when one cell of the upper row carried a taller box (its box union then grazed the row beneath, and a bare 0.5pt absolute overlap opened the fold). The vertical gate is now relative and local: a fragment must interpenetrate the TEXT band of BOTH cells flanking its gap by at least 20% of the smaller height (stackInterpenetrationMin). Measured inline stacks dip 25-56% into their line's band; separate rows overlap their neighbours' bands not at all, so table rows can never fold regardless of oversized boxes elsewhere in the row. maxStackFragmentCells's invariant is now documented. New regression test reproduces the reviewer's two-row interleave. Correctness: the pairwise cellReadingLess comparator switched rules per pair and was not a strict weak ordering (cycles possible over stacked geometry). Replaced with orderCellsForReading: columns pre-assigned by one left-to-right sweep, then a total-order sort by (column, top, left, index). Tests: the footnote-superscript and short-line negatives previously short-circuited before the new logic; both now reach it (height outlier present; multi-cell fold targets), and a gap-aligned-line negative was added. Removed now-caller-less paths: MergeFragmentedCells, the ExclusiveReextract option, GetTextByRectClaiming, and the char-band index that existed only for it. MergeFragmentedCellsExclusive is the single merge entry point. DPBench (200 PDFs, vs 513c968, same corpus/harness): errors 0 -> 0 extraction_accuracy 0.921572 -> 0.921895 (+0.000323) reading_order_nid 0.893762 -> 0.896273 (+0.002511) table_structure_teds 0.763126 -> 0.763126 (+0.000000, zero cases changed) heading_level_mhs 0.770814 -> 0.771969 (+0.001155) In-process latency (one process converting the 200-PDF corpus, best of interleaved runs): 5.969 -> 6.050 ms/page (+0.08 ms/page, ~+1.4%; iteration spread overlaps, machine under concurrent load). entropy.pdf: all seven audited corruptions remain fixed; inline fractions still read in place ("rate C H e symbols per second"). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Crossing runs must overlap another column's content core, not just the corridor midpoint: a wide entry in a ragged right-aligned numeric column sits alone in its own column and no longer counts as a crossing (review blocker; regression tests fail without the fix). - Raise the crossing floor to 3 lines so a merged two-line group header or a full-width title plus footnote can never fire the gate; near-miss negative tests added for both shapes, plus tests for dropGutterCrossingTables claim release and the token-fallback judgment. - Widen the degenerate-corridor width to half the modal character height (capped): corridors narrower than half a character cannot separate columns. - connect.go appendTableRows now reads a spanned cell's text only at its anchor slot, closing the same duplication bug render.Table had. - Filter wordCells through the marginal page-number split so the word path cannot re-swallow page numbers, and require vertical isolation before extracting a marginal number so in-table years/counts stay put. entropy.pdf: 19 -> 14 table blocks, 121 -> 66 rows, no duplicated rows, all real tables intact. DPBench: teds +0.00056, extraction +0.00039, errors 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ParagraphTextLine.FontSize reported the LARGEST cell size on a line, so a display equation carrying one oversized summation or integral measured as title-sized. The metric is shared by heading detection, body-font estimation, figure-label suppression and paragraph assembly, so the error propagated through all of them. Two levels carried the same maximum, and both are replaced: - buildParagraphTextLine now takes the character-count-weighted median of its cells' sizes (dominantFontSize). Invariant: a line's metric is the size at which the majority of its rendered characters are set; oversized decoration above the body (math delimiters, drop caps) and undersized annotation below it (super/subscripts, footnote and folio markers) are minorities of the glyph mass and may not define it. Weighting is by rune count, not box area, so one wide glyph cannot outvote the ordinary characters beside it. Exact ties resolve downward, the conservative reading for the prominence gates downstream. - mergeCellShell now takes its font size from the leading fragment, as it already does for font name, weight and colour. A merged run that absorbs one oversized glyph mid-word previously claimed that glyph's size for all its characters, which is why the median alone could not fix display equations. The majority rule is wrong here: a small-caps section title sets its capitals at the nominal size and its small capitals — the majority of characters — smaller, so the opening fragment is the correct representative. The new value is never larger than the old maximum. A declared font size is only used as evidence when it is commensurate with the glyph's rendered box (credibleCellFontSize). Some PDFs declare a fraction of a point inside a full-height box; a maximum ignored those harmlessly, a median would let them win. hasAlignedBodyLikeFollowingLine keeps the old maximum via linePeakMetric. It asks whether the following line is ordinary body prose, and a line that mixes body text with a materially larger run is structured content — a label/value table row — so the presence of any prominent run is the signal, not the majority. DPBench (200 docs): extraction 0.922288 unchanged, TEDS 0.763683 unchanged, errors 0. Reading order 0.896189 -> 0.895882 and heading MHS 0.771969 -> 0.770467, both from a single document whose 5pt table body was previously mis-measured as 6pt by the maximum; with the true body size a wrapped 6pt table-cell fragment now clears the font-prominence gate and is promoted. That is a heading-detection weakness the corrected metric exposes, not a new one it creates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every threshold in headings.go, figures.go and table/detect.go was chosen by a person looking at one document, which AGENTS.md warns against directly. Distil HURIDOCS pdf-document-layout-analysis labels into a gradient-boosted tree over docmill's own geometric features, compiled to Go. The existing rules survive as feature extractors; their cutoffs do not. Feasibility is already evidenced: HURIDOCS reaches high accuracy on Shannon 1948 in fast mode, which is LightGBM over PDF-derived features with no vision model. On the page docmill handles worst it correctly separates three Formula regions from the surrounding prose and keeps the page number out of them. Target architecture is a single classifier with no heuristic fallback, but removal is staged per class on measured evidence rather than up front. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Use github.com/dmitryikh/leaves rather than generating Go from the exported trees. LGEnsembleFromReader loads from memory, so go:embed gives both properties we wanted: no runtime disk dependency, and no exporter or tree walker to write. MIT, pure Go, no cgo, multiclass. Codegen is deferred rather than rejected; it would remove the dependency and make a decision path printable, which AGENTS.md asks for. The artefact is the same either way so the switch stays contained to one file. Add a Task 0 spike that proves the idea end to end on Formula alone before any of the full pipeline is built. A negative result there is cheap and should stop the project. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review fixes: classify-then-route pipeline (figures/tables are decided before lines exist today, so assembly becomes class-agnostic and routing happens on labels), a region model gating Table/Picture candidates (gutter persistence is a group invariant no line feature can express), DPBench held out of training entirely, an explicit 11-class label mapping, a transitional precedence rule for per-class migration, lexical features, class weights, pinned training reproducibility, and latency budgeted over feature extraction rather than tree walks alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Task 0 of the learned layout classifier plan: prove end to end that a gradient-boosted tree over docmill's line geometry can find display equations, before committing to the rest of the plan. The emitter assembles pages CLASS-AGNOSTICALLY — straight from raw text cells, no figure drops and no table carve-outs — because the equations this is meant to catch are exactly the lines today's pipeline swallows into fake tables. A dump from the default path would not contain them. Teacher labels come from HURIDOCS pdf-document-layout-analysis v0.0.35 in fast mode over 20 documents, joined to lines by containment fraction of the line box, with the teacher's integer page dimensions rescaled into docmill's before any overlap is computed. entropy.pdf is held out of training entirely; the remaining 19 documents are split by document, never by line. Go and Python agree to 8.5e-22 across the fixture (spike verify), which is also what justifies rewriting the model header's version=v4 to v3 — leaves reads only v2/v3, and the guard is a version check rather than a parser branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the limit On entropy.pdf, held out of training entirely, the model correctly labels 7 of the 12 display equations that docmill currently emits as Markdown headings, and claims zero genuine headings. The current heuristics get 0 of 12. That answers Task 0's question: the signal IS in docmill's geometry. Held-out documents score P=0.720 R=0.810 F1=0.762, and only 2.4 F1 points of that depends on the lexical features, so geometry carries it. Two findings change what Task 1 and Task 4 have to do: The teacher, not the model, is the binding constraint. HURIDOCS emitted zero Formula regions for 1207.7214 and one for 1706.03762 despite both being full of display equations, and hand-adjudicating a 24-line sample of entropy.pdf's false positives found ~42% of them to be equations the teacher missed. Task 1's verification step is now on the critical path. The residual errors are region-shaped, not line-shaped: 43% of false positives sit inside a Picture or Table region, and 43% of false negatives are sub-script fragments under eight characters. Both are what the REGION model in the cascade exists to fix — not a reason for more line features. Also recorded: leaves reads only model header v2/v3 while LightGBM has written v4 since 4.0, and the eval harness (eval.py) that reads the heuristics' verdicts back out of docmill's real Markdown rather than calling the unexported detectors in isolation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This branch is rebased onto main, but the measurements were taken on ivan/layout-classifier-plan, which carries the parser and line-assembly work. That work changes the unit of classification itself: entropy.pdf assembles into 2,335 lines there and 2,551 here, and docmill emits 71 headings there and 67 here. So the "12 of 71 headings are display equations" baseline does not exist on main and the harness will not reproduce the figures from this base. Say so in both the research note and the spike README rather than let a reader run it and quietly get different numbers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rned-layout-classifier
Answers Task 4's deferred codegen question with numbers instead of taste.
First, a correction to the premise: generated Go cannot be loaded INTO
leaves. leaves.Ensemble embeds an unexported lgEnsemble and exposes no
constructor beyond its parsers, so codegen replaces leaves rather than
feeding it.
The model makes this easy — every split is decision_type=2 and every tree
num_cat=0, so there are no categorical splits and the decision is
uniformly `if IsNaN(v) { v = 0 }; v <= threshold`. spike gen asserts both
and refuses to generate otherwise, rather than emitting Go that scores
differently from the trainer. shrinkage is ignored on purpose: LightGBM
bakes the learning rate into leaf_value, and leaves ignores it too.
Equivalence is the thing that makes the choice safe, so it is tested
hard: against leaves over 200,000 random vectors, including NaN in every
feature position, the worst difference is exactly 0.0.
Generated is ~33% faster (17.4µs vs 25.9µs), allocation-free, and drops
an 8ms / 3.3MB / 17.7k-allocation start-up parse. Interleaving the flat
arrays into 24-byte node records did NOT pay (17.8µs) — the walk is bound
by branch misprediction, not cache lines. Kept with the benchmark so the
experiment is not repeated.
It also closes the explainability gap the plan expected to have to record
against AGENTS.md: spike explain prints the decision path per line, which
leaves cannot do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…breaks Corpus size is not the scaling axis. Node count saturates against num_leaves, not data: 5 docs give 7,526 nodes and 19 docs give 8,448, at 94% of the per-tree cap. Going to 1000 documents can add ~6% more. Class count is the axis. Training the actual 11-class LINE model (plus a Background class) is a flat 10.75x: 3,600 trees, 90,823 nodes, 9.0 MB and 131k lines of generated Go. That still compiles, but costs 6.9s to rebuild after every retrain. The same trees as a go:embed'ed binary blob rebuild in 0.35s, produce an identical binary size and identical predict speed, and cost 4.4ms once at start-up. So the Task 4 recommendation refines: keep the flat-array walker and the equivalence tests, and swap Go literals for a packed blob when the model goes multiclass. Also notes that Go's build cache hashes content rather than mtime, so a touch-and-rebuild measurement reads as free and is meaningless. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scales the Task 0 spike from 20 documents to DocLayNet's 80,863 human-annotated pages (CDLA-Permissive-1.0). On 256,338 held-out lines the model reaches 0.776 accuracy / 0.744 macro-F1, and Formula — the class the spike existed to test — improves from 0.762 F1 to 0.803. Two things make this more than "more data". The teacher is gone. The spike's binding constraint was HURIDOCS' own recall, and DocLayNet's labels are hand-drawn by trained annotators, so that ceiling disappears. And the model stopped leaning on content: math_frac was the single largest feature by gain in the spike and does not appear in the top twelve here. Pure layout geometry takes the top seven places, which is what AGENTS.md asks for. We take DocLayNet's LABELS but not its features. Its pdf_cells column looks like a shortcut and is a trap: font.size is 1.0 for 29,135 of 29,180 sampled cells, there are no bold/italic flags, and cells are far more fragmented than docmill's. Training on those would be exactly the training/serving skew the plan warns about. So the pipeline extracts the 81,471 single-page PDFs and lets docmill compute its own features — 0 parse failures, 3,099,657 lines. The annotation join needs SEPARATE x and y scale factors: DocLayNet stretches every page into a 1025x1025 COCO square without preserving aspect ratio, so one shared factor displaces boxes by a large fraction of a line height. List-item is the weakest class at 0.588, and diagnostically so: it is distinguished by a leading marker and a hanging indent, and the spike feature set has neither. That is the case for building the full Task 2 vector rather than the spike's twenty. Emission is now parallel across documents — 81k pages in ~9 minutes — and skips unparseable files rather than abandoning the corpus. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…aseline Task 2 of the learned layout classifier plan, plus the emitter Task 1 needs. The feature vector now lives in pkg/pdf/layoutfeatures.go and grows from the spike's 20 to 32. Everything here MEASURES; nothing decides — the existing heuristics keep their verdicts, and this is the numeric view of the same signals so the model can weigh them jointly. Three of the new features are aimed at a diagnosed failure. List-item was the worst class on DocLayNet at 0.588 F1, and a list item is defined by a leading marker and a hanging indent, neither of which the spike measured. has_list_marker, numbering_depth and content_left_offset measure exactly that. The marker test is structural — the shape of the leading token — not a word list, and caption_marker matches "<word> <number>" so it works on Tabelle 2 as well as Table 2. LayoutDebugRows reports the feature vector AND what today's heuristics call each line, in one pass over the same class-agnostic assembly. That sameness is the point: Task 1's baseline and the model have to be scored on identical lines or the comparison means nothing. It replays the pipeline by BOX rather than by cell index, because orderCells renumbers every cell and an index captured before reading order no longer identifies the same cell after it. repeat_frac returns 0 for single-page documents rather than the naive 1/1. DocLayNet is 81k single-page PDFs, so the naive answer would train the model on a constant 1.0 that becomes a real varying signal at inference — training/serving skew of the quietest kind. The spike emitter now delegates here instead of carrying its own copy of the feature code, so there is one definition and the trainer reads the contract out of the binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…very class
The number the whole plan is gated on ("a class where the model does not
beat the heuristic is not a class to migrate") now exists. Scored on
DocLayNet's val split, 256,338 lines, with both sides graded on the SAME
class-agnostic lines and joined by the SAME rule.
Current heuristics: Table 0.327 F1, List-item 0.218, Picture 0.137,
Section-header 0.519, Text 0.674, and 0.000 on the five classes docmill
has no detector for. The model wins all eleven.
The fake-table defect is now measured rather than anecdotal: 31% of all
display-equation lines in DocLayNet are currently emitted as table cells,
along with 25% of Picture lines and 16% of Text.
List-item at 0.126 recall / 0.792 precision is the other shape worth
seeing — DetectStructure is right when it fires and fires on one list
line in eight. That is conservative detection working as AGENTS.md asks,
paying in recall.
The Task 2 feature vector (32 vs the spike's 20) lifts accuracy 0.776 ->
0.805 and macro-F1 0.744 -> 0.762, and List-item by 0.073 — three times
the next class, which is exactly what the new marker/indent features
targeted. Diagnosis confirmed: the bottleneck was features, not model,
corpus or hyperparameters.
Lexical ablation costs 3 accuracy points and 2 macro-F1, so the model
degrades gracefully without content features as AGENTS.md requires.
Page-header is actually BETTER without them (0.742 -> 0.826).
Recorded with the caveat that matters: this measures agreement with
DocLayNet's taxonomy, not extraction quality. docmill's zero on
page-furniture is a design choice, not a defect, and line-level Table and
Picture scores describe a region decision. Formula is the exception that
needs no argument — 0.812 against no detector at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 5. ExtractionOptions.ClassifyThenRoute selects an alternate path in reroute.go; the default path is untouched and the flag is off. The restructure exists because today's order decides tables BEFORE lines exist — DetectTables consumes cells directly and its cells never reach the assembler — so a line classifier bolted onto today's assembly cannot take over those decisions. New order: assemble all cells class- agnostically, route each line, hand each destination to its existing builder. Every routing decision is still the existing heuristics'; this isolates refactor risk from model risk. Table detection moved into a shared detectPageTables used by both paths, so "the reroute changes the ORDER, not the decisions" is enforced by there being one definition rather than asserted in a comment. GATE PASSED: byte-identical on all 200 DPBench documents. That is strictly stronger than matching DPBench scores, since every metric is a function of this output. Five unit tests pin the same property on prose, lists, tables, multi-page and empty pages so regressions fail go test. Being precise about what the gate proves: byte-identity is partly by construction. The rerouted path computes the class-agnostic line set but still rebuilds prose blocks from routed CELLS, deliberately — routing lines directly would have conflated the refactor's risk with the plan's stated hazard that class-agnostic assembly creates lines which never existed before. So the harness also measures that hazard: 52 of 6137 lines (0.847%) straddle a routing boundary, i.e. are only partly inside a table or heading region. Those are the lines whose output changes the moment Task 6 routes lines instead of cells. The samples are column-adjacency artefacts — two headings merged into one line — not a systemic assembly failure. Shadow mode (SetShadowRouteSink) records the destination of every class-agnostic line, changing nothing, nil in production. It becomes the live confusion matrix once the model is embedded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tasks 4 and 6. Display-equation lines wrongly emitted as table cells fall from 34.0% to 7.6% — a 78% reduction — while genuine table lines lose one line in 197 and DPBench stays byte-identical. The 12-class DocLayNet model ships as a 2.93 MB packed blob rather than generated Go source: identical binary size and identical prediction speed, but 0.35s to rebuild after a retrain instead of 6.9s, for a 4.4ms decode at start-up. Nothing parses LightGBM text at run time. Go and Python agree exactly — the fixture replays LightGBM's own RAW scores for 48 vectors stratified across all twelve classes, worst delta 0. Raw rather than probabilities so a mismatch points at the tree walk instead of being blurred by softmax. The loader refuses a model whose feature count differs from LayoutFeatureNames, which is the guard against a model that runs fine and is confidently wrong. The migration rule is a plurality vote, not a threshold: a candidate table is rejected when the most common label among its lines is Formula. That is argmax over the region's line-label distribution — the region feature the plan describes — so no hand-picked constant enters. A migration that introduced a tuned cutoff would be self-defeating. Nothing is deleted yet; the heuristic still proposes every candidate and the model only vetoes. DPBench cannot measure this class, and that is itself the finding. The veto never fires on any of its 200 documents — DPBench is financial reports, patents and manuals, and equation-as-fake-table is a scientific paper problem. So DPBench proves no collateral damage, and the evidence that the migration HELPS comes from DocLayNet's scientific_articles split against human labels. Each remaining class needs its metric chosen before it is migrated: a class DPBench cannot see is not a class DPBench can approve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g it A byte-identical result cannot distinguish "the rule ran and correctly found nothing" from "the rule never ran". SetFormulaVetoSink observes every candidate table the veto considers, so the difference is now evidence rather than inference. On DPBench the veto evaluates 47 candidate tables across 200 documents and rejects none. The plurality label of the lines inside them is Table 32 times, Picture 12, Background 2, Section-header 1 — never Formula. On two scientific papers the same code evaluates 125 candidates and rejects 41. So the no-op on DPBench is a property of the corpus, not a broken flag. It also surfaces something we are NOT acting on yet: the model disagrees with the table detector on 15 of those 47 DPBench candidates, thinking they are pictures or page furniture. That is a preview of what migrating the Table class would change, and a reason to measure it carefully. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…vanvanderbyl/docmill into ivan/learned-layout-classifier Signed-off-by: Ivan Vanderbyl <ivanvanderbyl@users.noreply.github.com>
LearnedRouting moves every line-class decision to the learned classifier, not just the Formula veto. Measured on 2,334 DocLayNet val pages spanning all six document types, against human labels: Picture 0.111 -> 0.692 (+0.581) Section-header 0.515 -> 0.739 (+0.223) List-item 0.161 -> 0.194 (+0.033) Text 0.652 -> 0.672 (+0.020) Table 0.272 -> 0.280 (+0.008) weighted F1 0.504 -> 0.569 (+0.065) Picture is the headline. The hand-tuned figure filter had 0.959 precision at 0.059 recall — it fired on one figure line in seventeen. The model trades a little precision for ten times the recall. The substitutions are surgical. isHeadingLine is replaced by one model call and everything downstream survives — level assignment, adjacent merging, marker attachment — because the plan keeps heading LEVELS (the teacher is flat on headings) and those stages build blocks rather than classify. Same for lists: the model replaces isListBlockCandidate and the run-context rule that required a neighbouring list item, which is what held DetectStructure to 0.126 recall. Block labels come from a plurality vote over the class-agnostic lines inside the block, not the best-matching line. A three-line paragraph contains three lines at containment 1.0, so "best match" would pick one arbitrarily and the label would depend on iteration order. Table barely moves (+0.008) and that is expected: whether a region is a table is not a line decision, and the region model is not built yet. List-item is held back by the RENDERER, not the model, which scores 0.661 standalone. rewriteListItem only rewrites a line that literally starts with a marker glyph, so a list item the model recognises by geometry alone cannot be expressed as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…into ivan/learned-layout-classifier
The learned path has been reachable only from the spike binaries. This puts it behind `docmill convert -learned-layout <input.pdf>`, so the work can be run against real documents today. The flag sets ClassifyThenRoute, LearnedRouting and LearnedFormulaRouting together. The last one is not redundant: reroute.go gates the Formula veto on LearnedFormulaRouting alone, so LearnedRouting by itself would hand headings, lists and figures to the model while formulas kept flowing into tables — not what the flag says it does. Without the flag, convertOptions returns exactly the option set ExtractMarkdown builds, so the default conversion is unchanged. Parsing now goes straight to the FlagSet rather than through stripArgSeparator: "--" is the flag package's own terminator, so `convert -- <path>` keeps working and a path beginning with "-" starts working. Not built or tested — this workspace has no Go toolchain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merging the parser and line-assembly work back onto this branch changed the unit the model classifies: entropy.pdf now assembles into 2,335 lines where the model was trained on 2,551, and the corpus went from 3,099,657 lines to 3,136,911. The model was scoring lines that no longer exist. That is the training/serving skew this project keeps guarding against, arriving through a merge rather than through a feature. Re-emitted all 81,471 pages, rejoined and retrained. The better assembler makes a better model, which is a good sign for the assembly work itself: accuracy 0.8054 -> 0.8362, macro-F1 0.7624 -> 0.7738 on the DocLayNet val split. Routing measured again over 2,334 pages spanning all six document types: Picture 0.110 -> 0.714 (+0.604) Section-header 0.491 -> 0.775 (+0.284) Text 0.716 -> 0.738 (+0.022) List-item 0.215 -> 0.231 (+0.015) Table 0.352 -> 0.365 (+0.014) weighted F1 0.556 -> 0.629 (+0.072) Model artefact and fixture regenerated together, so the Go/Python agreement test still pins the model actually shipping rather than the one it replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
List-item routing was stuck at 0.132 recall while the model scored 0.661 standalone, and the cause was my own plurality vote rather than the renderer. A four-line list item carries its marker on the first line only. The other three read as ordinary prose and the model labels them Text, so the vote went 3-1 against and the item rendered as a paragraph. That accounted for 88% of the list items the routing missed. "Is this block a list item?" is really "does this block START a list item?", so it now asks the topmost line. Second change: when the model recognises a list item whose marker is not a character we can strip — a glyph the font maps oddly, or one drawn rather than typed — render it as a list item anyway. Refusing because no literal bullet is present is the renderer overruling the classifier. List-item 0.231 -> 0.630 (+0.400), recall 0.132 -> 0.569 Text 0.738 -> 0.748 (+0.010) weighted 0.629 -> 0.679 (hand-tuned baseline 0.557) Precision falls 0.905 -> 0.707 for four times the recall, which is the right trade for a class the heuristics were finding one time in eight. Nothing else moves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
19,447 lines and ~1.9 MB of artefacts removed: the HURIDOCS teacher labels, the binary Formula model, the leaves runtime and the Go-source codegen. All of it answered "does this work at all", and it did; the answers live in docs/research and the code is in git history. What replaced each piece: human DocLayNet labels instead of a model teacher, a 12-class model instead of binary, and a packed blob in pkg/pdf instead of 12k lines of generated Go. Keeping two model runtimes and two label pipelines around after that would just be two things to keep in sync. github.com/dmitryikh/leaves drops out of go.mod entirely — the version=v4 header workaround goes with it. The one capability worth saving is ported rather than deleted: ExplainLineClass now walks the SHIPPING model's trees and prints why a line got its label. That is the AGENTS.md explainability requirement, and the plan expected to have to record it as a gap because leaves exposes only scores. Also documents the retraining rule the merge taught us: a line-assembly change changes the model's input, so it invalidates the model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ometry The check that decides whether table-structure learning is viable at all, run before building anything on top of it. FinTabNet ships what we need and DocLayNet did not: per-cell bounding boxes, the grid as HTML with colspans, the table's own box, and a filename that joins straight to a real PDF page. 16 GB, CDLA-Permissive-1.0, and the licence explicitly covers computational use. Coordinates are PDF points with a BOTTOM-left origin against docmill's top-left, so every box needs flipping by the page height. Third corpus, third different coordinate convention — hence checking against real extracted text rather than assuming. Result: 270 of 300 tables align, 86.4% of cells matched. The first attempt read 53%, and the fix is worth recording because it would otherwise look like bad data. docmill's TextCells are whole-LINE rects, so a table row is frequently a single cell that cannot sit inside any one annotated cell. Matching against WordTextCells instead moved it from 53% to 86.4% without touching the transform. The geometry was right all along; the granularity was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The layout classifier answers "is this region a table". This answers "where are its columns" — the part a class label cannot express and the part TEDS actually scores. Today it is hand-tuned gutter thresholds. pkg/table/columngaps.go enumerates every plausible column boundary in a region and describes each with 16 numbers: gap width against the table and against a character, persistence across rows, alignment counts either side, vertical ruling coverage, density and digit share. Same pattern as the line features — keep the measurement, drop the cutoff. It decides nothing; the existing detector keeps its verdicts. Candidates deliberately over-generate. Every text-free x-interval is proposed, including wide spaces inside sentences, because a boundary never proposed can never be chosen. Truth comes from FinTabNet's HTML grids. Walking the token stream reconstructs which row and column every cell occupies; only cells spanning exactly ONE column bound that column, because a colspan=3 cell sits across two internal boundaries and says nothing about where either lies. Including them would smear columns together — which is the error the hand-tuned rule makes on merged headers. 61,801 grids reconstructed with zero parse failures. Trained on 184,900 candidates from 39,446 tables, held out on FinTabNet's own val split: per boundary precision 0.865 recall 0.887 F1 0.876 per table 65.4% recover EVERY boundary correctly The per-table number is the honest one: a table with eight of nine boundaries right still has the wrong grid, and TEDS will score it wrong. Not yet wired into the structure builder, so TEDS is unchanged at 0.7637. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Task 6 step 5: "If the model loses on that class, KEEP the heuristic and record why." It loses. The model is good — 0.876 F1 per boundary on FinTabNet's val split, 65.4% of tables recovered exactly. On DPBench it is a net regression: mean TEDS over all cases 0.857837 -> 0.852837, and documents scoring zero go 18 -> 19. So -learned-columns exists and is off by default. The headline TEDS is UNCHANGED at 0.763683, and that is the part worth remembering. One document goes from a perfect table to nothing, 1.0000 -> 0.0000, and the summary row absorbs it without moving because it averages over a fixed documents-with-tables set that this one falls outside. On this harness an unchanged headline TEDS does not mean unchanged table quality; check case_results. Why it loses: the regressed document is a display equation beside body text. The heuristic makes no table; the model makes a four-column one. That is the fake-table defect this project exists to remove, reintroduced. The cause is a category error in the wiring — the model was trained only on regions that ARE tables, so it has never been asked "is this one?" and answers the question it was given with confidence on a region that has no columns at all. Letting the heuristic decide WHETHER a region has columns and the model only decide WHERE did not fix it, so the mechanism is further downstream. Not worth chasing before the region model exists, which is now the second class to ask for it. Also adds pkg/gbm, one model runtime shared by pkg/pdf and pkg/table. pkg/table cannot import pkg/pdf, so without it each would carry a copy of the decoder and they would drift. Default path byte-identical over all 200 DPBench documents; vet and the full suite clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two measurements demanded this stage. Handing table routing to the line model moved Table F1 by 0.008, and the learned column model turned a display equation into a four-column table because nothing had asked "is this a table at all". Both are region questions: whether a run of lines is a table depends on gutter persistence, column-count stability and row regularity across the whole run, and no feature of any line expresses them. GroupLineRegions collects maximal runs of same-label lines; regions.go describes each with 32 region-scoped features. Every one is a property of the RUN — if it could be computed from one line, the line model already had it. Among them is the distribution of line labels inside the candidate, so "80% of these lines scored Formula" is now a feature and the equation-versus-table arbitration is learned rather than ordered. Gutter persistence reuses ColumnGapCandidates, so the region stage and the column stage agree about what a gutter is. Trained on 1,088,757 candidates from all 81,471 pages, joined by IoU — correct here where it was wrong for lines, because candidate and teacher boxes are both region-shaped and match near one-to-one. Offline on DocLayNet val it is strong exactly where the line model is weakest: Picture candidates are correct 5.6% of the time and the gate sorts them at 0.782 precision / 0.710 recall; Table candidates 11.8%, gated at 0.671 / 0.688. End to end it currently does almost nothing: wired as a Picture gate it rejects 24 of 3,543 figure-label lines and moves Picture F1 by -0.002. The offline rate is over ALL candidates, most of them one or two lines that never become a block, so by the time a block reaches the gate the line path has already dropped the weak cases. The model is good and it is gating the wrong class. Table is where the gain is — 0.680 on candidates that are 11.8% correct — and it cannot be applied until table routing is model-owned. That is the next piece, and it also unblocks -learned-columns, whose one bad failure was a region with no columns at all. Default path byte-identical over 200 DPBench documents; vet and suite clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous note read "Table candidates are correct only 11.8% of the time"
as the gate's opportunity. It is a symptom. Accept/reject scores a candidate
overlapping a real table at IoU 0.49 exactly as wrong as one overlapping
nothing, and those demand opposite fixes.
Measuring the ceiling instead: only 42.8% of real tables have any same-class
candidate at IoU >= 0.5, against 78.4% reachable from contiguous line runs.
The gate cannot accept what nobody proposed, so 35.6 points sit on the floor
regardless of how good it gets.
Three separate causes, three different fixes:
- GroupLineRegions emits maximal SAME-LABEL runs, so one mislabelled line
splits a table and neither piece nor their union is ever offered. Using
teacher labels the ceiling equals the oracle exactly, so the whole gap is
line-label noise. Merging adjacent candidates alone recovers 24.3%.
- 39.8% of Picture regions contain no assembled line at all, and pkg/page
exposes no image placements and no path geometry. A text-line cascade
cannot see a photograph; this is a parser change before it is a model one.
- Page-header, Caption and Title ceilings are capped by line assembly
merging across column boundaries — two side-by-side captions become one
line. Out of scope for this project, and now measurably in the way.
Adds the three diagnostics and the note; no shipping code changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The interpreter is a port of PDFium's core/fpdfapi/page layer and already
walked every drawing operation, computing a bounding box for each. It then
discarded most of them, on one deliberate line:
// Image (and other) subtypes are out of scope for text extraction.
That line is why 39.8% of DocLayNet's Picture regions are invisible to us:
they contain no text, and pkg/page offered no non-text geometry at all.
Ported against pdfium 0db284a42 rather than written from the spec:
- Images, from Do on an /Image XObject and from inline BI/ID/EI. A PDF
image occupies the unit square under CTM * mt_content_to_user_, so the
box needs no image data.
- Shadings, from sh: no geometry of their own, they flood the clip or the
page box. Mesh shadings are not intersected with their coordinate data,
since the stream is never loaded.
- Filled paths. AddPathObject emits when stroked OR filled; only n paints
nothing. Every rule drawn as a thin filled rectangle was being lost.
- Rectangles as path starts. The run was only entered on m, so a path
written as re alone was invisible — and `re W n` is the commonest clip
in real PDFs.
- Clipping. W/W* arm the clip; it merges when the paint operator arrives,
AFTER the object is emitted, so a path is never clipped by the clip it
establishes. A degenerate path clips to empty rather than to nothing.
Form XObjects take their /BBox as an initial clip, and pointedly do NOT
inherit the enclosing one — that rides on the FormObject, and
DrawnObjects intersects it down into the children.
ClipPath keeps intersected bounding boxes, which is what PDFium's own
GetClipBox computes. The approximation errs toward keeping ink that a
non-rectangular clip hides, never toward dropping ink that is drawn.
`docmill render` writes one PNG per page outlining every object, or -json
for measurement. It exists because a matrix composed in the wrong order or
an unflipped y-axis yields numbers that look plausible in a dump and are
obvious on a page.
Text extraction is untouched: the clip is recorded on each object and
applied only by DrawnObjects, and RulingSegments filters to stroked paths so
filled shapes cannot invent a grid. All 200 DPBench documents produce
byte-identical Markdown against a binary built from a clean HEAD worktree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds `spike drawn`, which dumps every object the interpreter says a page draws, and ceiling_ink.py, which recomputes the region ceiling over ink instead of over assembled text lines. 2,806,188 objects from all 6,489 DocLayNet val pages in 24 seconds, zero failures. Against the annotations at IoU >= 0.5: Picture 22.1% -> 69.6% (ink alone 63.1%) Formula 62.0% -> 69.0% Table 85.8% -> 87.7% everything else unmoved, <= 1.8% from ink 58.3% of picture regions are matched by a SINGLE image XObject, so more than half need no clustering at all — the box is read straight off the content stream. Clustering supplies the rest. The two sources are complementary rather than competing: ink adds nothing to the text classes and text adds 6.5 points on top of ink for Picture, so the proposer needs both. Formula gaining 7 points was not the goal — its radicals and fraction bars are drawn rather than typed, so the ink cluster catches formulas the text run splits. Page-header stays at 26.8%. That is the line-splitting defect from the previous note, and no amount of ink addresses it. Clustering is done by rasterising and taking connected components. The obvious pairwise union-find does not finish: a chart is thousands of path operations in a small area, which is simultaneously the quadratic worst case and the exact input this measurement exists to handle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ines
GroupLineRegions emitted one candidate per maximal run of lines sharing a
PREDICTED label, so one misread line destroyed the only candidate there
was. Three replacements, each measured on DocLayNet val before being kept:
- Ink clusters. Candidates from what the page draws, now that the
interpreter reports it. Rasterise and flood-fill: pairwise union-find
over box proximity does not finish, because a chart is thousands of
path operations in a small area. Single objects are candidates in
their own right, since 58.3% of pictures are one image XObject.
- A label-free grouper. Atomic groups split on vertical gaps and
horizontal disjointness alone, then every contiguous merge of them.
The correct extent only has to be among the proposals; the model picks
it out, so label noise no longer destroys candidates.
- Column-gap line splitting, for lines that straddle two columns.
Region recall, old proposer -> new:
Section-header 54.5% -> 86.1% Picture 7.5% -> 71.1%
Table 42.8% -> 83.3% Caption 42.1% -> 70.4%
Page-footer 73.5% -> 82.9% Text 19.7% -> 69.6%
Formula 42.5% -> 78.1% Footnote 31.1% -> 67.6%
List-item 8.8% -> 72.8% Page-header 19.9% -> 44.7%
overall 34.7% -> 72.2%, at 375 proposals per page
Three measurements contradicted the design and changed it:
Neither split granularity is right. Coarse gives Table 81.4% / List-item
50.3%; fine gives List-item 66.8% / Table 76.9%. DocLayNet annotates each
list item separately, so coarse swallows a list; a table cut into thirty
fine groups needs thirty merges to rebuild. Running both levels recovers
each, and costs LESS than fine alone because dedup removes the overlap.
Persistence alone cannot find a running header. `Chapter 3 ... Page 45`
has body text beneath it spanning the corridor being tested, so
corroboration always fails — Page-header moved 0.1 points. Accepting a gap
of six ems without corroboration took it to 44.7%, Caption to 70.4% and
Formula to 78.2%.
Splitting lines costs Table 5.6 points, because a table row IS
column-separated cells. So the proposer runs on both line sets and offers
candidates from each, which returns Table to its best measured 83.3% with
every splitting gain kept.
InkProposals and SplitColumnLines are off by default and nothing routes
through the proposer yet. All 200 DPBench documents byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous region model was a GATE: candidates arrived already carrying a
class, because they were built by grouping lines that shared a predicted
label, so it only had to answer "keep it?". That framing is why it could
never fix an extent — a table proposed one line short and the same table
proposed correctly are each plausible in isolation, and a gate only ever
sees one at a time.
The new proposer splits on geometry and clusters ink, so a candidate
arrives with no opinion about what it is. Assigning the class is therefore
part of the job, and the useful side effect is a confidence comparable
ACROSS candidates — which is what makes non-max suppression possible over
the ~375 proposals a page now produces.
- pkg/pdf/proposalfeatures.go: 55 features. Shape, isolation (whitespace
above and below, which nothing previously expressed), internal
structure, the line model's label distribution, and ink counts — "one
image" and "four hundred paths" are different kinds of region.
- pkg/pdf/nms.go: SelectRegions. Containment is checked both ways, not
just IoU: a line inside an accepted table has tiny IoU with it but is
not a separate region, and a confident heading inside a sprawling
low-confidence merge must suppress the merge rather than the reverse.
- pkg/gbm: PredictProbabilities, since ranking candidates needs every
class on one scale rather than only the winner's probability.
- proposalmodel.bin: 4200 trees, 12 classes, 55 features, 6.62 MB.
Trained on 3.03M proposals, held out on 1.99M. Background is 87.6% of the
raw data and does not fit in memory, so the subsampling is selective
rather than uniform: near misses (IoU 0.25-0.5 against a real region) are
kept in full and only candidates overlapping nothing are thinned. Those
near misses are exactly what suppression must rank below the correct
extent, and uniform sampling would discard them at the same rate as the
easy ones.
Per-candidate precision is low by construction — many overlapping
candidates share a class — which is the number suppression exists to fix
and not a quality result. The end-to-end decomposition is measured
separately by eval_regions.py, which matches greedily one-to-one so
duplicate detections of one table cannot each score a true positive.
Nothing routes through this yet; LearnedProposals is off by default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ProposalFeatures runs once per candidate and the proposer offers ~375 candidates per page, so the per-call cost is the number that looks harmless and the per-page cost is the one that decides whether this can ship. ProposalFeatures, one candidate 52 us ProposalFeatures, a whole page 291 ms ColumnGapCandidates, page-wide box 722 us 291 ms/page against a plan budget of "a rounding error on the current 12-14 ms/page". The cost is ColumnGapCandidates, whose inner loop is gaps x rows x cells within the candidate — cheap for one candidate, quadratic-ish in aggregate once every candidate pays it. Recording the number before deciding what to do about it. The fix is likely to redefine the gutter features at PAGE scope — find the gutters once, then ask only whether each is clear within a candidate's own rows — which is arguably the better feature but changes the contract and needs a retrain, so it waits on the end-to-end quality result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Running the same 783 val pages WITH and WITHOUT suppression located the fault precisely. Without it, recall sat near what the proposer allows: Table 0.741, Page-header 0.696, Picture 0.582. With it, every class roughly halved: 0.281, 0.350, 0.264. The right candidate was being found and classified correctly, and then discarded. Two causes, both mine. Class weights destroyed the calibration suppression depends on. Weighting by sqrt(background/count) upweights Table 4.5x and Title 45x — right for balanced classification, wrong when suppression RANKS by probability. The classifier called 44 candidates per page Table against 0.73 real tables per page. Removing the weights roughly doubled per-candidate precision everywhere: Table 0.296 -> 0.514, Section-header 0.354 -> 0.557. Class probability cannot rank extents. A table one line short and the correct table have nearly identical content features, so they score nearly the same and suppression picks between them close to randomly. The fix is an IoU head — a second model over the same 55 features predicting how well an extent matches what it overlaps — with Rank now class probability x predicted overlap. It needed no new data: the IoU was already in the joined dataset. It reaches 0.119 mean absolute error, and its top three features are gap_above, gap_below and height_frac, which is exactly the signal the classifier had no way to use. Removing the weights also exposed a latent crash. With too little signal to split on, LightGBM emits trees with NO splits — a single constant. The packer recorded a root at node N while the tree contributed no nodes, so N pointed one past the end of the array and inference read off it. Always present, never triggered, and it crashes rather than answering wrongly. Fixed in both the packer and the reader, with tests that build tiny models by hand. Same 783 pages, weighted F1 0.2032 -> 0.2508: Table 0.260 -> 0.461 Text 0.145 -> 0.195 Section-header 0.247 -> 0.324 Page-header 0.346 -> 0.230 Picture 0.254 -> 0.314 List-item 0.126 -> 0.063 Page-footer 0.487 -> 0.549 Table nearly doubled; List-item halved. The stage under-segments flowing text — one merged candidate wins and then suppresses, by containment, every paragraph inside it. That is a plausible decomposition of a page, just not the one DocLayNet annotates. Recorded rather than papered over: the next question is whether the IoU head is weak on Text or whether containment suppression is wrong WITHIN a class, and the same with/without experiment will separate them. Nothing routes through this; LearnedProposals stays off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PredictProbabilities read scores[best] inside the loop that overwrites the slice. Once i passes best, scores[best] is exp(0)=1, so every later class subtracts 1 instead of the maximum. With small raw scores that is the bad version of wrong: finite, plausible, and silently distorting every probability after the winning class — every ranking measurement taken through this function is suspect. With large raw scores it is loud: exp(raw-1) overflows to +Inf, the sum is Inf, and the probabilities come back 0 for the winner and NaN for the rest, which is what finally surfaced it — the JSON emitter refused the NaN 12 documents into a 800-page run. The hunt also caught two infrastructure traps now fixed in goenv.sh: the Go module cache lived in tmpfs and half-vanished (the sandbox blocks proxy.golang.org, so missing zips meant unbuildable, not slow), and one `go build | head` pipeline masked that failure and left a stale binary that emitted an entire diagnosis dataset without the field under diagnosis. Caches now live on disk, GOPROXY falls back to the host's intact download cache, and the regression tests build tiny models by hand so none of this needs a 5 MB artefact to reproduce. Raw scores in the tens of thousands also mean the classifier is training with LightGBM's default lambda_l2=0 — saturated leaves. Noted for the next retrain rather than smuggled into this fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three changes, one story: the region stage was being scored through a broken softmax, on a biased sample, with a decision rule that let the majority class win by default. Fixed in that order, it scores weighted F1 0.5525 end to end — not the 0.25 previously recorded. The decision rule: every candidate is now scored as its best REAL class, and survives only when real-class probability x predicted IoU >= 0.2. Background still vetoes, but through the product — a candidate that is probably nothing has a low class score, a low predicted overlap, or both — instead of winning the argmax outright, which had been discarding candidates whose runner-up was correct: 22.6% of all tables, 16.4% of list items. The threshold was swept OFFLINE (0.5399 argmax / 0.5448 at 0.1 / 0.5525 at 0.2 / 0.5469 at 0.3) in a Python simulator of SelectRegions that matches the Go path to four decimal places on identical input; the Go re-run reproduced 0.5525 in every per-class row. The sampling lesson is the expensive one. All previous spot measurements used the FIRST 800 pages of DocLayNet val, which is not shuffled: Text is reachable for 33.7% of regions there against 69.6% corpus-wide. The old and new proposers reach IDENTICAL regions on those pages — nothing ever regressed, the sample was skewed — and the same pipeline scores 0.29 on the head sample and 0.54 on a seeded random one. rand800.txt (seed 20260806) is the sample from here on. Queued for one combined retrain rather than three: lambda_l2 (raw scores reach +/-1.6e6 and saturate the probabilities NMS ranks with), page-scope gutter features (291 ms/page vs a 12-14 ms budget), and table near-miss rebalancing for the IoU head, whose errors concentrate on same-class wrong-extent tables. LearnedProposals stays off; nothing routes through the region stage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oU head The three fixes queued for the single combined retrain, built together because each one alone would invalidate the others' retrained artefacts. Page-scope gutter index. The old gutter features called doctable.ColumnGapCandidates once per candidate: 722us a call, ~375 candidates a page, 271 of the 291 ms/page this stage cost. The index does the page-wide work once — cells grouped into rows, per-row x-spans merged and sorted — and each candidate sweeps a 2pt occupancy grid over the rows it spans, the same discretise-don't-compare trick that saved the ink clustering. Feature extraction drops 291 ms -> 16 ms per page, measured by the same benchmark that caught the problem. The feature names keep their contract slots but their values shift (rows are the index's own grouping now), which the count-based contract check cannot catch — the blobs and the extractor must move in one commit, and do. One test fixture turned out to be a lesson in disguise: words placed at identical x every row read as five persistent gutters, and that is CORRECT — aligned corridors are a table grid. Prose has to stagger, and the fixed fixture does. lambda_l2=10 in both trainers. With LightGBM's default of zero, the unweighted rare classes saturate: leaf values explode until raw scores reach +/-1.6 million and the softmax collapses to exact 0s and 1s — destroying precisely the ranking granularity non-max suppression needs, and incidentally detonating the softmax aliasing bug as Inf/NaN. Table-weighted IoU head. The end-to-end diagnosis found the head's errors concentrated where they cost most: when a table is outranked by a same-class wrong extent, the head prefers the wrong one 80% of the time. The join now records `near` — the class of the best-overlapping region at ANY IoU, which is exactly what a Background-labelled near-miss erases — and the IoU trainer weights near-Table rows 3x. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The combined retrain — lambda_l2=10, page-scope gutter features, and the
table-weighted IoU head, trained together because each alone invalidates
the others' artefacts — lifts the region stage from 0.5525 to 0.6275
weighted F1 on the seeded random val sample. Every class improved;
precision now runs 0.6-0.87 everywhere. Feature extraction is 16 ms/page
against the previous 291.
Two user-facing surfaces expose the stage on real documents:
docmill render -regions [-json] one labelled box per kept region
docmill convert -region-markdown the model owns the page
Region-routed Markdown is the plan's end state in miniature: the region
stage decomposes the page, tables run the existing grid machinery only
inside model-approved boxes, picture innards are dropped, headings and
lists come from region classes, and unclaimed lines fall through as
paragraphs so nothing silently vanishes. Missing models degrade to the
routed path, never to an empty page.
Testing it surfaced "the output dropped a lot of characters and words",
which decomposed into three different things once measured:
- Table markup, not content: char retention 90.7% while WORD retention
was 98.6%, and the worst document had zero missing words. A table
rendering as prose loses pipes and padding — structure loss wearing a
content-loss costume. Char counts cannot tell them apart.
- Two real bugs, fixed: only the first table inside a region was
rendered, and cells inside the region but outside the detected grid
were dropped. Both now re-attach.
- Misclassified pictures, guarded: a Picture region with NO ink in it —
no image, no path, no shading — is a misclassification wearing a
green box, and deleting a paragraph on its say-so is not the model's
call to make. Text-only Picture regions keep their text.
Remaining word loss (~1.3%) is chart legends inside genuine figures,
dropped by design per the annotation standard.
Default path byte-identical over all 200 DPBench documents against the
pre-region baseline binary. Vet and the full suite clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three defects, each visible in one held-out document, each fixed at the
renderer rather than papered over in a model:
- Prose now goes through the SAME paragraph assembler as the default
pipeline. The first version joined line texts with spaces, and the
seams showed: "Vol. 27, pp." came out "Vol . 27 , pp ." and
hyphenated line breaks lost their hyphens. One text builder, both
paths; the leftover pass also inherits paragraph splitting instead of
reimplementing it with atomic groups.
- A Picture region no longer eats its caption. The figure box
over-reached by one line and silently deleted "Fig. 1 — Schematic
diagram of a general communication system". Member lines the LINE
model calls Caption are rescued and rendered as prose; the region
model owns the figure, not the caption's deletion when a second model
disagrees. The five diagram box labels (INFORMATION SOURCE,
TRANSMITTER...) still drop, by design.
- Mathematics no longer becomes a heading. "log2 M log10 M log10 2"
carried Section-header votes from BOTH models — short, centred,
isolated is the shape of a title, worn by maths, and geometry cannot
tell them apart. But a third of its characters are digits and no real
heading reads like that: a would-be heading whose digit+math fraction
exceeds 0.2, or whose lines the line model calls Formula, renders as
prose instead. A renderer guard, not a classification — the text
stays, only its promotion into the outline is refused.
On the Shannon sample the region path now produces the identical heading
outline to the default pipeline, keeps the caption, and loses exactly
five words — the diagram's internal labels. DPBench retention holds at
98.7% words / 91.6% chars, the remainder being figure innards and table
markup accounting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five DPBench rounds, each fix found by reading per-case failures rather
than tuning. v1 -> v5, against the tuned pipeline's 0.92/0.90/0.76/0.77:
extraction 0.93 -> 0.92 (per-case 6 wins / 2 losses — still ahead)
MHS 0.58 -> 0.77 (parity)
TEDS 0.65 -> 0.71
NID 0.84 -> 0.86
Every headline regression turned out to be wearing a costume:
- The worst "reading order" case had nothing out of order — a table of
contents rendered as dot-leader prose instead of the default's TOC
table. Region prose now goes through the same assembleWithToc.
- "Table structure" was a starved detector: the region stage FOUND the
unruled tables and rendered zero rows, because detection ran inside
the region box while the anchored detector's "Table 1:" anchor lives
in a neighbouring region. Detection now runs PAGE-WIDE with full
context and the model's Table regions decide which detections are
accepted — the division the plan called "the region model owns table
acceptance", arrived at by benchmark rather than argument.
- "Heading hierarchy" was three defects: caption regions promoted to
headings (one false heading on a heading-free doc scores 1.00 -> 0.00);
a first guard that reused the marker-SHAPE feature as a hard rule and
deleted "Activity 1:" headings wholesale (right as a feature, wrong as
a rule — it now names actual caption words); and the region
classifier's 70% heading recall (each miss on a one-heading doc is
another 1.00 -> 0.00). Headings are now the UNION of the heuristic
detector and the region classifier, with the caption and math guards
applied to the OUTPUT — attached to any single detector, the guards
had a path around them within one measurement round.
Proposer experiments, both measured on the seeded random sample: wider
merge spans are a dead end (+25% proposals for +0.1% reach — Text is near
its assembled-line ceiling). Decisive-gap 6 -> 4 is kept (+0.7pp overall,
Title +16.7pp, at +4% proposals) with the train/serve note that serving
models were trained on 6.0-emitted features; fold into the next retrain.
Remaining, named in the research note: NID -0.04 (21 cases, unproven
cause), TEDS -0.05 (balanced per-case), 247 ms/page inference cost.
Default path spot-checked byte-identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first net-negative commit of the project: -2,014 / +389, and the +389
is mostly docs/LEARNINGS.md — the consolidated record of everything this
project paid to find out, indexed so nobody pays twice.
Deleted, each with its measurement:
- The old REGION GATE (regions.go, regionmodel.go/.bin, gateRegions,
-learned-regions, LayoutRegionRows, spike regions + its trainers).
Superseded by the proposal pipeline; as a Picture gate it rejected 24
of 3,543 figure lines for -0.002 F1. Its one real discovery — Table
acceptance is where the value is — now lives in the region-routed
path as page-wide detection with model acceptance.
- The LEARNED COLUMN stack (columnmodel.go/.bin, columngaps.go,
-learned-columns, SetColumnDerivation instrumentation, colcheck and
tablegaps tools). 0.876 F1 on FinTabNet val, net regression on
DPBench (0/0/0/-1 per-case), and it once turned a display equation
into a four-column table. deriveColumnBoxes carries the epitaph and
the resurrection conditions: PubTabNet-class breadth AND the
acceptance gate in front. The FinTabNet reconciliation scripts stay
as reference; fintabcheck stays (dataset-geometry checker, still
useful).
- Line-model HEADING ownership in the routed path. DPBench: MHS 0.63
vs the heuristic's 0.77, 39 documents worse, 17 better. Per the
plan's own rule, a class the model loses stays with the heuristic,
recorded. The model keeps lists, figure innards and the Formula veto
— the classes it measurably wins.
Flags are now two: -learned-layout and -region-markdown.
Shared helpers (unionBoxes, stability, meanOf, maxOf, varianceOf) moved
to proposals.go with the surviving consumers.
Default path byte-identical over all 200 DPBench documents. go vet and
the full suite clean.
Also adds docs/research/2026-08-08-path-to-winning.md: the evidence-based
route to a learned pipeline that beats every remaining option — close
NID (-0.04, 21 cases, diagnose-first), fix the acceptance veto so absence
of a region cannot kill a detection (Table region recall is 0.548; the
fake-table set is the regression gate), get under 100 ms/page, one
retrain carrying the queued skew; then word-primitive proposals for the
line-capped classes and PubTabNet table structure; then the flip and
class-by-class deletion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Final three-way verification after the deletions: -learned-layout now matches or beats the default on every DPBench headline metric (0.93 / 0.90 / 0.76 / 0.77 at 67 ms/page; per-case NID +7/-2, MHS +1/-0, TEDS 0/0). The heading revert also fixed its TEDS deficit — the line-model heading pass had been stealing cells from tables. Region path unmoved by the cleanup, exactly as intended. Path note updated: the bar to beat is now -learned-layout, and it is the interim recommendation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.