Skip to content

fix(html): extract definition lists instead of discarding them - #4502

Merged
cragwolfe merged 1 commit into
Unstructured-IO:mainfrom
r0h1tb:fix/html-definition-lists
Sep 27, 2026
Merged

cragwolfe merged 1 commit into
Unstructured-IO:mainfrom
r0h1tb:fix/html-definition-lists

Conversation

@r0h1tb

@r0h1tb r0h1tb commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

<dl>, <dt> and <dd> are mapped to RemovedBlock, so partition_html() drops definition lists and everything inside them. Documentation gets hit hardest: Sphinx renders every documented function, class and attribute as a <dl>, so partitioning an API reference page returns the headings and nothing else.

partition_html(text=(
    '<h1>API reference</h1><dl class="py function"><dt>connect(host, port=5432)</dt>'
    "<dd><p>Open a connection to the database server.</p></dd></dl>"
))
# [Title('API reference')]

Glossaries and key/value lists disappear the same way, with no error.

Fix

Map <dl> to ListBlock and <dd> to ListItemBlock, which the parser already anticipates: the ListBlock docstring says "maybe a <dl> element at some point", and the list-depth code already counts dl ancestors for dd items. <dt> becomes a plain BlockItem, so a term gets its type from its text like any paragraph. The v2 (ontology) parser already keeps definition lists.

Tests

Two new tests in test_partition.py: a glossary (exact element types, text and category_depth) and a nested Sphinx-style API reference (all text kept, in order). Without the fix they return [('Title', 'Glossary', 1)] and ['API reference']. The html, chunking, documents, md, text and email tests go from 1201 passed on main to 1203, with no failures.

This touches the same mapping table as #4451 (<details>/<summary>), so whichever lands second will need a small rebase.

Review in cubic

<dl>, <dt> and <dd> were mapped to RemovedBlock, so partition_html()
silently dropped glossaries and every Sphinx-generated API reference,
where each documented function, its parameters and its return value
live in a <dl>.

Map <dl> to ListBlock and <dd> to ListItemBlock, as the ListBlock
docstring and the list-depth code already anticipate, and <dt> to a
plain BlockItem.

@cragwolfe cragwolfe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SAFE TO MERGE at reviewed head 89f382c4799a1c0d991123d1141bdab879f5f044 against main at 1bedf7be0bea9db5c5dc3a9a2dab83078b4db54d. No actionable Critical or High findings remain.

The changed tags reuse existing traversal classes; the parser already counts <dl> ancestors for <dd> depth. The added tests cover exact glossary element types and depth, plus nested Sphinx text retention in source order. All six required exact-head checks passed at review time.

(authored by codex)

@cragwolfe
cragwolfe merged commit ddf4453 into Unstructured-IO:main Sep 27, 2026
53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants