fix(html): keep text inside unrecognized elements

[huuufu]

Summary

Fixes #4496. Resolves #3842 (#3708 was closed in its favor).

The v1 HTML parser gives every tag it has no class for a DefaultElement, and DefaultElement emits only the element's tail: its own text and everything inside it are dropped without a trace. Before the parser rewrite in #3218 (0.15.0) the legacy parser listed font in its TEXT_TAGS, so for <font> this is also a regression. In practice this loses:

• <font>, still common in email HTML (Outlook and Gmail reply headers).
• Inline XBRL, required in SEC 10-K/10-Q filings: <ix:nonFraction> wraps the numbers inside sentences, and <ix:nonNumeric>/<ix:continuation> wrap whole notes, tables included.
• Custom elements (<my-card>, the <hauptteil_AB> from bug/content within custom HTML tags is skipped #3708), <ins>, <nobr>, <ruby>, <picture>.

Everything routed through partition_html() is affected: .html, .md, .eml/.msg HTML bodies, .epub, .rst, .org.

Measured on the repo's fixtures. "Token coverage" is the share of word tokens from a reference text, built from each fixture's own HTML, that appear in the output; the reference approximates, but is not, what a browser displays (method in Testing).

fixture	elements, main → this PR	token coverage, main → this PR
example-10k.html (same for -utf-16)	314 → 466	66.0% → 100.0% of 27,619 tokens
example-10k-1p.html	23 → 24	97.3% → 100.0%
example-steelJIS-datasheet.html (same for -utf-16)	12 → 19	75.0% → 88.7%
eml/email-no-utf8-2008-07-16.062410.eml, HTML body	2 → 22	31.5% → 93.8%

On main the 10-K loses its cover-page values (form type, fiscal year end, registrant name, shares outstanding) and all of Notes 1–14. The email keeps two quoted paragraphs; the replies and their From:/Sent: headers are inside <font>. The other 40 HTML-routed fixtures in example-docs produce identical output.

Change

DefaultElement drops its two overrides and inherits Phrasing.iter_text_segments() and Flow.iter_elements(), so an unrecognized element is transparent.

On the block-vs-inline question in #3842, no guess is needed. The element keeps identifying as phrasing, which is how a browser renders an unknown element (display: inline), and the parser already turns each block nested inside phrasing into its own element, as for a <p> inside a <span>. So <font> around two words joins the sentence, and <ix:nonNumeric> around a note with headings and tables yields the same elements as the note without the wrapper. Checked on the real filing: example-10k.html with every ix: wrapper stripped (content kept) partitions to the same 466 elements; 448 are identical, and the other 18 are tables whose text differs only in whitespace from TableBlock code this PR doesn't touch. Text mixed with blocks (<x>before<p>para</p>after</x>) and emphasis and link annotations come out exactly as they do inside a <span>.

Elements whose content is not rendered as text now have to be skipped explicitly. They are registered as RemovedPhrasing, which behaves exactly like the old DefaultElement, so their output does not change:

• metadata: head, title, and legacy xml data islands (such as Office document properties);
• fallback content: object, iframe, audio, video, canvas, noembed, noframes;
• form-control internals: select, datalist, textarea;
• dialog, which is hidden until opened and on static pages is mostly popup UI;
• svg, math, and Inline XBRL's hidden ix:header (facts, contexts and units that filings put in a display:none <div>).

Each entry has a test that leaks text if the entry is removed. Judgment calls:

• math stays skipped so this doesn't pre-empt fix: preserve MathML content in HTML partitioning #4478, which adds a Math class. Whichever PR lands second deletes that line; ruff's F601 flags the duplicate key otherwise.
• dialog is skipped whether or not it has open, which keeps today's output for both. An open-aware class would be a small follow-up if you want open dialogs included.
• embed, source and track are left transparent: libxml2 nests the following content inside these void elements (<p>See <embed src=x> for details.</p> puts "for details." inside <embed>), so skipping them would keep losing that text.
• Tags that are already registered (button, label, details, dl, figure, form, nav, template, …) are untouched.

Flow._page_number now walks ancestors in a loop instead of recursing, with the same result. DefaultElement is a Flow, so a block inside N nested unknown elements now looks up N ancestors while the phrasing traversal is itself about 2N frames deep; without this, a <p> inside 250 nested unknown elements raised RecursionError.

The v2 parser (html_parser_version="v2") is not affected; it keeps unknown-tag text as UncategorizedText.

Before / after

from unstructured.partition.html import partition_html

for html in [
    "<p><font face=Arial>Quarterly revenue</font> rose 12%.</p>",
    '<p>Revenue was $<ix:nonFraction name="us-gaap:Revenues">4,213</ix:nonFraction> million.</p>',
    "<my-card><h2>Pricing</h2><p>Plans start at $10.</p></my-card>",
]:
    print([str(e) for e in partition_html(text=html)])

Before:

['rose 12%.']
['Revenue was $ million.']
[]

After:

['Quarterly revenue rose 12%.']
['Revenue was $4,213 million.']
['Pricing', 'Plans start at $10.']

Testing

• test_parser.py: the three DescribeDefaultElement tests that pinned the old behavior now assert transparency. New tests cover blocks inside <foobar>, <ix:nonNumeric> and nested <my-card><font> (same elements and metadata as without the wrapper), text mixed with a block (same as inside <span>), annotations, each newly skipped tag (parametrized, including open and closed <dialog>, plus head via a body-less document), and the page number of a block nested in 250 unknown elements.
• test_partition.py: end-to-end tests for <font> and for Inline XBRL (inline fact, block ix:nonNumeric note with a table, ix:header content that must not appear).
• Against main's code, all 36 new or changed parser tests and both end-to-end tests fail.
• test_unstructured/partition/html/: 402 passed, 1 failed on this branch; 367 passed, 1 failed on main. The failure is the same test on both, from my Windows checkout's CRLF fixtures.
• ruff check . and ruff format --check on the changed files pass.

Other suites compared on main and this branch

Compared test-by-test; every test present on both has the same outcome.

suite	main	this branch
test_md.py / test_email.py	39 / 72 passed	39 / 72 passed
test_msg.py	49 passed, 1 failed	49 passed, 1 failed
test_epub.py + test_rst.py + test_org.py	46 passed, 1 failed	46 passed, 1 failed
chunking/test_title.py	46 passed, 7 skipped	46 passed, 7 skipped

The failures are the same tests on both sides and come from my environment (missing image/PDF extras, CRLF fixtures).

How token coverage was measured

The reference is each fixture's HTML (for .eml, the HTML body partition_email() uses) parsed with lxml's HTML parser, with <head>, <script>, <style>, <noscript>, <template> and hidden/inline display:none elements removed, split into \w+ tokens. Coverage is the share of reference tokens, counted as a multiset, found in the concatenated element text. It does not apply stylesheets, so it is an approximation of what a browser shows.

No fixture gains output tokens that aren't in the reference, and I read every newly emitted element. Every token still missing on this branch was already missing on main: one-character page numbers (skipped by design), text in a few table cells, and an antivirus footer the email places after </html>.

Limitations and not verified

• ix:header is matched by its conventional ix prefix, because libxml2's HTML parser has no namespaces. A filing that binds the Inline XBRL namespace to another prefix would have its hidden header emitted. The repo's filing fixture uses ix.
• Other XML vocabularies embedded in HTML, such as Confluence ac: macros, now contribute their text like any unrecognized element. I have no such fixture.
• As with <span>, adjacent unknown elements with no whitespace between them are joined: <x-a>foo</x-a><x-b>bar</x-b> gives foobar, which is also how a browser renders them.
• The azure/spring-weather.html ingest fixture, whose source isn't in the repo; test_auto.py doesn't collect in my environment.

[chrikrah]

@huuufu approving. Your parser and test changes merge into today's main (74e2fea) on top of #4502's dl registration. On Linux nothing fails in either suite, there or on the branch:

$ TMPDIR=<beside the exported trees> python -m pytest test_unstructured/partition/html/test_parser.py test_unstructured/partition/html/test_partition.py -q
265 passed               # merge base 0ca5563
300 passed               # head 7a0efd0: 38 test ids added, 3 removed
38 failed, 262 passed    # head tests with parser.py from 0ca5563: the 38 added ids, exactly
267 passed               # main 74e2fea
302 passed               # main 74e2fea plus this change, files byte-identical to git merge-tree's result
# the trees sit under /tmp, and with the default TMPDIR is_temp_file_path() skips one test's last-modified lookup
# Python 3.13.15, lxml 6.1.3, libxml2 2.14.6

An <img> inside <picture>, <font> or a custom element now comes out as an Image, and a <pre> inside a custom element as a CodeSnippet. The next paragraph then gets parent_id=None instead of the heading. A bare <img> or <pre> already does that on main, since HIERARCHY_RULE_SET (partition/common/metadata.py:58) lists neither category under Title:

$ python hier_table.py <0ca5563 tree> <7a0efd0 tree> <74e2fea tree> <74e2fea plus this change>
partition_html(text="<h1>Section</h1>" + X + "<p>After</p>")
cell: what comes out between the Title and "After" / the parent of "After"
X                                                        0ca5563               7a0efd0
<picture><img src=a.png alt=Diagram></picture>           - / Section           Image / None
<p><font><img src=a.png alt=Diagram></font> caption</p>  Text / Section        Image Text / None
<my-card><img src=a.png alt=Diagram></my-card>           - / Section           Image / None
<my-code><pre>x = 1</pre></my-code>                      - / Section           CodeSnippet / None
<img src=a.png alt=Diagram>                              Image / None          Image / None
<pre>x = 1</pre>                                         CodeSnippet / None    CodeSnippet / None
74e2fea gives the 0ca5563 column: True | 74e2fea plus this change gives the 7a0efd0 column: True

example-10k-230p.html, which your table leaves out, changes like example-10k.html, and no changed fixture gains an Image or a CodeSnippet. Every example-docs file of a type that routes through partition_html(), at 0ca5563 and at 7a0efd0:

$ DEPS=<markdown 3.10.2 and pypandoc-binary 1.17, the uv.lock pins> python sweep.py <0ca5563 tree> <7a0efd0 tree>
.html .htm .eml: 58 files, 49 identical, 7 changed, 2 raise the same error on both
   314 -> 466  example-10k-230p.html, example-10k-utf-16.html, example-10k.html
    23 -> 24   example-10k-1p.html
    12 -> 19   example-steelJIS-datasheet-utf-16.html, example-steelJIS-datasheet.html
     2 -> 22   eml/email-no-utf8-2008-07-16.062410.eml
   Image or CodeSnippet elements gained by the 7 changed: 0
   the 2 errors with process_attachments=False: identical
.md .epub .rst .org .rtf: 17 files, 17 identical, 0 changed, 0 errors   # pandoc 3.9
.msg, attachments off and on: 10 runs, 10 identical, 0 changed, 0 errors

@cragwolfe in example-docs this changes only the 10-K, steelJIS and one email fixture. Could you review it?