fix(html): extract <details>/<summary> content instead of discarding it

[r0h1tb]

Problem

<details> and <summary> were both mapped to RemovedBlock, so partition_html() discarded the whole subtree. On the FAQ page from the issue, every question and answer disappears and the caller gets a shorter element list rather than an error:

>>> partition_html(text="<h1>Support FAQ</h1>"
...   "<details><summary>Which Bluetooth profiles are supported?</summary>"
...   "<p>Your device supports A2DP and HFP profiles.</p></details>")
[<Title 'Support FAQ'>]          # both the question and the answer are gone

Fix

<details> is an ordinary block container, so it becomes Flow, like <div>.

<summary> gets its own Heading subclass. Its label is emitted as a Title, including when it follows an expand/collapse icon or sits in a <p>/<div> wrapper, so chunk_by_title() opens a new section at each question and the question stays with its own answer.

The label has no heading level of its own, so its category_depth is one below h6, plus one per enclosing <details>. That nests it under whatever heading precedes it, ends it at the next heading of any level, and nests an inner <details> label under its parent's.

<summary> was also listed twice, under both phrasing and the removed-form group; there is now one mapping.

Tests

New tests in test_partition.py cover extraction, chunk sections, a <details> with no summary, nesting, icon- and wrapper-led labels, annotations and page number on the label, and parent_id/category_depth for a heading before and after a <details> and for nested <details>. The hierarchy tests fail without the depth change (parent_id is None).

partition/html, partition/common, documents and chunking: 1114 passed before, 1117 after, 40 skipped both times. test_common.py and chunking/test_basic.py can't be collected here without the extras, same on main. Ruff is clean.

Rebased on main, version 0.27.19.

[cubic-dev-ai]

No issues found across 4 files

<sub>Shadow auto-approve: would auto-approve. Bug fix in HTML parsing: previously dropped <details>/<summary> content is now extracted, with tests pinned; low-risk mapping change.

Re-trigger cubic</sub>

[cragwolfe]

High — a leading image or block wrapper makes a <summary> lose its Title boundary.

unstructured/partition/html/parser.py:1023 maps summary to Heading, but Heading supplies Title only to the initial text/phrasing run in Flow.iter_elements() (parser.py:365-376). Since img is a non-phrasing ImageBlock, <summary><img src="/chevron.svg" alt="">Second question?</summary> first emits the image, then processes Second question? through the generic tail path without Heading._ElementCls; it becomes Text, not Title. A <div>/<p>-wrapped question follows the same child-block path. As a result, chunk_by_title(..., combine_text_under_n_chars=0) never sees the second FAQ boundary and can merge that question/answer into the preceding section.

Please add summary-specific traversal that preserves the title role for the summary label across leading images and ordinary block wrappers, while keeping the details body and post-summary tail outside the title and avoiding a duplicate title for a nested heading. The regression should assert exact element types/order and two separate chunks; covering link/emphasis and page metadata on the label would also protect the annotations carried by this path.

(authored by codex)

[r0h1tb]

Good catch, thanks. Fixed in 51dabe8, and rebased onto main since the branch had gone stale.

<summary> now has its own element class that emits every run of label text as a Title, including text after a leading image like the chevron and text inside plain <p>/<div> wrappers (recursively). Images, a nested <h3> and other specialized blocks keep their own types, so a nested heading still produces exactly one Title. The summary's tail and the rest of the <details> body are still handled by the parent, so they stay outside the title.

New tests assert the exact element types and order plus two separate chunks for the icon case, and cover the <p>/<div> wrappers, a nested heading, text after the summary, and emphasis, link and page-number metadata on a wrapped label. The html, chunking, documents, md and text tests go from 1129 passed on main to 1139 on this branch, with no failures.

[cragwolfe]

Finding

• [P1] Preserve heading hierarchy for summary-generated Titles — unstructured/partition/html/parser.py:475-483. Summary._iter_label_elements() now emits every summary run as Title, but _ElementAccumulator._category_depth() maps a non-h1–h6 Title to depth 0. The shared set_element_hierarchy() stack then treats each summary as a top-level heading. I reproduced this on the exact head 51dabe8436708a28cb1713a90ad7ca00f78805b9: <h2>FAQ</h2><details><summary>Q1</summary><p>A1</p></details><h2>Shipping</h2><p>x</p> produces Q1 with parent_id=None instead of FAQ, and incorrectly assigns Shipping under Q1; nested Outer/Inner summaries are also siblings instead of parent/child. This silently regresses parent_id/section hierarchy for existing headings and affects downstream HTML/ontology/RAG consumers.

Please fix the hierarchy semantics for summary-derived Titles, preserving real heading levels and making nested details/summary relationships explicit, then add regression assertions for parent IDs and category depths (including a heading after a details block and nested details). The current extraction and focused text/type tests do not catch this.

Verification: uv run --locked reproduction against the exact PR head confirmed the incorrect parent IDs; all reported exact-head CI checks were green, but CI does not cover this hierarchy case.

(authored by codex)

[r0h1tb]

Fixed in fbd8e9b. A summary label now gets category_depth one below h6, plus one per enclosing <details>, so it takes the heading before it as its parent and ends at the next heading. In your example Q1's parent is FAQ and Shipping is top-level again, and nested Outer/Inner labels come out as parent/child (depths 6 and 7). Added parent_id/category_depth assertions for both cases and for a wrapped label.

One thing I left alone here: a leading <img> in the summary (the chevron case) still drops the heading from the parent chain, because Image isn't a valid child of Title in HIERARCHY_RULE_SET. That's the same issue you flagged on #4503, so I'll handle it there instead of in both PRs.

Also rebased on main, now 0.27.19.