Skip to content

Optimize XLSX subtable detection memory usage - #4357

Open
CyMule wants to merge 11 commits into
mainfrom
xlsx-sparse-connected-components
Open

CyMule wants to merge 11 commits into
mainfrom
xlsx-sparse-connected-components

Conversation

@CyMule

@CyMule CyMule commented Jun 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replace worksheet-sized NetworkX graph construction with traversal over populated cells. Preserve four-neighbor connectivity, row-overlap merging, element ordering, text, HTML, and metadata. NetworkX remains a test dependency for comparison with the previous algorithm, and is removed from the XLSX runtime extra.

Compatibility

No new file-size, row, cell, or processing limit is introduced. Parsing, populated-cell semantics, and table rendering are unchanged. Exact serialized output matches the old algorithm for generated multi-sheet workbooks across header/HTML options and six existing XLSX fixtures. This is evidence of compatibility, not a guarantee for every possible workbook. Pandas still materializes worksheet data and a boolean mask; memory is reduced, not capped.

Validation

Synced with current main on October 1, 2026 (released as 0.27.15).

  • XLSX suite: 88 passed, including all 512 possible 3x3 occupancy patterns compared with the legacy NetworkX algorithm.
  • Exact serialized-output parity: six existing workbooks and generated multi-sheet workbooks with four header/HTML combinations.
  • Ruff and lock consistency checks passed.
  • Fresh component-detection benchmark, three repetitions per case:
Case Sparse seconds / additional sampled-peak MB Dense seconds / additional sampled-peak MB
Dense table 0.0100 / 6.4 0.0318 / 19.1
Sparse wide edges 0.0035 / 1.9 5.8608 / 1573.5
Separated blocks 0.0034 / 0.2 0.8578 / 200.7

These measure component detection, not total API latency. Peak RSS is sampled every 5 ms, so it is a lower bound on the true peak.

@CyMule
CyMule marked this pull request as ready for review October 1, 2026 17:15

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic

Comment thread scripts/performance/benchmark_xlsx_connected_components.py Outdated
Comment thread scripts/performance/benchmark_xlsx_connected_components.py
Comment thread scripts/performance/benchmark_xlsx_connected_components.py Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.

Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.

Re-trigger cubic

@CyMule
CyMule marked this pull request as draft October 2, 2026 03:33
@CyMule
CyMule marked this pull request as ready for review October 2, 2026 04:12
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 7 files

Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.

Re-trigger cubic

TrueFurina pushed a commit to TrueFurina/unstructured that referenced this pull request Oct 2, 2026
An Ubuntu package download in Unstructured-IO#4357's CI took 54 minutes for 262 MB,
compared with 7–39 seconds on the other runners. The existing retry loop
waits indefinitely for a command to return, so it cannot recover from a
download that continues extremely slowly.

This change shares dependency setup across the unit, extras, and ingest
jobs. It applies 30-second HTTP/HTTPS inactivity timeouts, APT retries,
and three bounded five-minute attempts for index updates and downloads.
Packages are downloaded first and installed with `--no-download`,
keeping dpkg installation outside the outer timeout. Setup appears as a
separate step, and the PPA's implicit index refresh is replaced with an
explicitly bounded refresh.

The package set, Tesseract version check, and test commands are
preserved. Persistently poor networking now produces a clear setup
failure instead of an hour-long download; the change does not guarantee
a faster mirror.

Validation: Bash syntax, ShellCheck, shfmt, workflow YAML parsing, and
whitespace checks passed. A mocked harness verified success, retry after
timeout, exhaustion, and installation sequencing. A disposable Ubuntu
24.04 container validated the APT flags with strict index update,
download-only fetching, and offline installation of libmagic-dev. The
full helper also completed successfully in a disposable Ubuntu 24.04
ARM64 container, including LibreOffice, the Tesseract PPA, Korean OCR
data, and diffstat; Tesseract reported 5.5.1. The repository
release-version check passed on Ubuntu. GitHub CI passed on aa97013,
including the unit, dependency-extras, ingest and Dockerfile jobs that
use the new helper; no end-to-end CI speedup has been established.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant