Skip to content

feat: add clean_newline to concatenate words wrapped across lines - #4462

Open
TrueFurina wants to merge 9 commits into
Unstructured-IO:mainfrom
TrueFurina:feat/clean_newline
Open

TrueFurina wants to merge 9 commits into
Unstructured-IO:mainfrom
TrueFurina:feat/clean_newline

Conversation

@TrueFurina

@TrueFurina TrueFurina commented Aug 28, 2026 •

Copy link
Copy Markdown

Summary

Per #2513: when text is wrapped, a word that continues on the following line is often written as a character followed by a dash and whitespace. Adds a clean_newline function that removes the hyphen and whitespace and joins the two parts.

Changes

  • unstructured/cleaners/core.py: new clean_newline(text, pattern=r"(\w+)-\s+(\w+)") using re.sub(pattern, r"��", text), exactly as proposed in the issue
  • test_unstructured/cleaners/test_core.py: parametrized tests covering hyphen+newline, hyphen+space+newline, plain text passthrough

Verification

  • All 5 parametrized cases pass locally (docu-
    ment → document, docu-
    ment → document, hello-
    world → helloworld, no-hyphen text unchanged, word-
    word → wordword)
  • ast.parse passes on both files
  • Implementation matches the issue's proposed signature/docstring intent

Review in cubic

@mi7ak2020-stack

Copy link
Copy Markdown

D

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants