fix(partition): read spooled uploads in place instead of copying them by CyMule · Pull Request #4419 · Unstructured-IO/unstructured

CyMule · 2026-07-23T02:50:22Z

What

Partitioning and file-type detection read a SpooledTemporaryFile upload in place. They no longer copy the whole document into a BytesIO first.

Before: upload spool (disk) → BytesIO copy in detect_filetype → BytesIO copy in DOCX/PPTX/PDF/TSV/XML partitioner
After:  upload spool (disk) → read in place by both

Two related robustness fixes come with it, since removing the detector copy exposes both:

Input	Before	After
`detect_filetype(file=<rolled-over spool>, metadata_file_path="a.md")`	`.name` is an int fd, so extension lookup raises `TypeError` (previously hidden by the copy)	`.md` taken from `metadata_file_path`
`convert_to_bytes(tempfile.TemporaryFile())` or any file-like wrapper	`ValueError: Invalid file-like object type` (text-encoding detection, PDF page rendering)	bytes, stream rewound

A spool passed to detect_filetype() still comes back at position 0, as before.

Why

Python 3.11+ (this package's minimum) gives SpooledTemporaryFile the full buffered-I/O interface that zipfile, pandas and the detectors need. The BytesIO copies were a <3.11 workaround, and each one is an extra heap allocation the size of the document. Fixing it here removes the need for callers such as unstructured-api and core-product to wrap uploads in proxy objects to avoid the copy.

Release

Bumps to 0.27.12. 0.27.11 is already released.

Impact

54.9 MB DOCX through the partition API: peak process RSS −51.2 MiB, peak container memory −52.6 MiB, median latency 9.343s → 9.197s.

Validation

New regression tests for in-place detection (in-memory and rolled-over spools), the non-string-name fallback, and convert_to_bytes on generic streams. All fail without this change.
Before/after comparison over file_utils, partition/common, DOCX, PPTX, text, XML, TSV, MD, HTML and auto suites: no new failures. The one existing test that read the detection context after closing the spool now asserts inside the with block.
Ruff check/format pass.

Remove the remaining SpooledTemporaryFile-to-BytesIO copy from detect_filetype(), keeping the spool rewound afterward. Ignore non-string file names when deriving the extension so a rolled-over spool falls back to metadata_file_path, and let convert_to_bytes() read any seekable binary stream. Release as 0.27.11 since 0.27.10 is already tagged.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic}

cubic-dev-ai

0 issues found across 2 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic}

cubic-dev-ai

0 issues found across 3 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve. This PR does not meet the repository auto-approval settings.

Re-trigger cubic}

…se it

cubic-dev-ai

All reported issues were addressed across 7 files (changes from recent commits).

_{Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic}

chatgpt-codex-connector · 2026-10-02T04:13:00Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

cubic-dev-ai

4 issues found across 14 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="test_unstructured/file_utils/test_filetype.py">

<violation number="1" location="test_unstructured/file_utils/test_filetype.py:517">
P2: These assertions make the new tests Unix-only: Windows rolled spools expose a string `.name`. Use a platform-neutral fixture/name assertion, or explicitly gate this Unix-specific check, so the supported Windows test suite does not fail.</violation>
</file>

<file name="CHANGELOG.md">

<violation number="1" location="CHANGELOG.md:5">
P3: `detect_filetype()` returns a `FileType`, not the input stream, so this says the wrong value is returned. Say the spooled file is left at position 0.</violation>

<violation number="2" location="CHANGELOG.md:7">
P3: This promises that every object with `read()` and `seek()` is rewound, but `BytesIO` and path-named `BufferedReader` inputs take separate branches that do not rewind the stream. Narrow this statement to the newly handled generic file-like streams.</violation>
</file>

<file name="unstructured/file_utils/filetype.py">

<violation number="1" location="unstructured/file_utils/filetype.py:515">
P2: Rolled spools on Windows expose a truthy temporary-path string without an extension, so this early return bypasses `metadata_file_path` and loses the supplied file type. Return the stream extension only when it is non-empty, then fall through to the metadata path.</violation>
</file>

_{Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic}

cubic-dev-ai · 2026-10-02T04:17:07Z

+    with tempfile.SpooledTemporaryFile(max_size=1) as spooled_file:
+        spooled_file.write(b"# Heading\n\nSome *markdown* text.\n")
+        assert spooled_file._rolled
+        assert not isinstance(spooled_file.name, str)


P2: These assertions make the new tests Unix-only: Windows rolled spools expose a string .name. Use a platform-neutral fixture/name assertion, or explicitly gate this Unix-specific check, so the supported Windows test suite does not fail.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At test_unstructured/file_utils/test_filetype.py, line 517: <comment>These assertions make the new tests Unix-only: Windows rolled spools expose a string `.name`. Use a platform-neutral fixture/name assertion, or explicitly gate this Unix-specific check, so the supported Windows test suite does not fail.</comment> <file context> @@ -489,6 +491,41 @@ def test_it_detects_EMPTY_from_empty_file_like_object(): + with tempfile.SpooledTemporaryFile(max_size=1) as spooled_file: + spooled_file.write(b"# Heading\n\nSome *markdown* text.\n") + assert spooled_file._rolled + assert not isinstance(spooled_file.name, str) + + assert detect_filetype(file=spooled_file, metadata_file_path="notes.md") == FileType.MD </file context>

cubic-dev-ai · 2026-10-02T04:17:07Z

-                return os.path.splitext(file.name)[1].lower()
+            # -- a temporary file (including a rolled-over `SpooledTemporaryFile`) can have an
+            # -- integer file-descriptor as its name, which carries no extension.
+            if isinstance(name := getattr(file, "name", None), (str, os.PathLike)) and name:


P2: Rolled spools on Windows expose a truthy temporary-path string without an extension, so this early return bypasses metadata_file_path and loses the supplied file type. Return the stream extension only when it is non-empty, then fall through to the metadata path.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At unstructured/file_utils/filetype.py, line 515: <comment>Rolled spools on Windows expose a truthy temporary-path string without an extension, so this early return bypasses `metadata_file_path` and loses the supplied file type. Return the stream extension only when it is non-empty, then fall through to the metadata path.</comment> <file context> @@ -511,8 +510,10 @@ def extension(self) -> str: - return os.path.splitext(file.name)[1].lower() + # -- a temporary file (including a rolled-over `SpooledTemporaryFile`) can have an + # -- integer file-descriptor as its name, which carries no extension. + if isinstance(name := getattr(file, "name", None), (str, os.PathLike)) and name: + return os.path.splitext(name)[1].lower() </file context>

Suggested change

if isinstance(name := getattr(file, "name", None), (str, os.PathLike)) and name:

if isinstance(name := getattr(file, "name", None), (str, os.PathLike)) and name:

if extension := os.path.splitext(name)[1]:

return extension.lower()

cubic-dev-ai · 2026-10-02T04:17:07Z

+
+- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0.
+- **Use `metadata_file_path` when a file's name is not a path**: a temporary file, including a rolled-over `SpooledTemporaryFile`, can report an integer file descriptor as its `.name`. File-type detection now ignores non-string names and falls back to `metadata_file_path` for the filename extension instead of raising `TypeError`.
+- **Read any seekable binary stream in `convert_to_bytes()`**: streams such as `tempfile.TemporaryFile()` or file-like wrappers previously raised `ValueError("Invalid file-like object type")` from text-encoding detection and PDF page rendering. Any object with `read()` and `seek()` is now read from the start and rewound, including a `BufferedReader` whose `.name` is not a path (an in-memory buffer or a file descriptor), which previously raised `AttributeError` or closed the caller's descriptor.


P3: This promises that every object with read() and seek() is rewound, but BytesIO and path-named BufferedReader inputs take separate branches that do not rewind the stream. Narrow this statement to the newly handled generic file-like streams.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 7: <comment>This promises that every object with `read()` and `seek()` is rewound, but `BytesIO` and path-named `BufferedReader` inputs take separate branches that do not rewind the stream. Narrow this statement to the newly handled generic file-like streams.</comment> <file context> @@ -1,3 +1,11 @@ + +- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0. +- **Use `metadata_file_path` when a file's name is not a path**: a temporary file, including a rolled-over `SpooledTemporaryFile`, can report an integer file descriptor as its `.name`. File-type detection now ignores non-string names and falls back to `metadata_file_path` for the filename extension instead of raising `TypeError`. +- **Read any seekable binary stream in `convert_to_bytes()`**: streams such as `tempfile.TemporaryFile()` or file-like wrappers previously raised `ValueError("Invalid file-like object type")` from text-encoding detection and PDF page rendering. Any object with `read()` and `seek()` is now read from the start and rewound, including a `BufferedReader` whose `.name` is not a path (an in-memory buffer or a file descriptor), which previously raised `AttributeError` or closed the caller's descriptor. + ## 0.27.11 </file context>

Suggested change

- **Read any seekable binary stream in `convert_to_bytes()`**: streams such as `tempfile.TemporaryFile()` or file-like wrappers previously raised `ValueError("Invalid file-like object type")` from text-encoding detection and PDF page rendering. Any object with `read()` and `seek()` is now read from the start and rewound, including a `BufferedReader` whose `.name` is not a path (an in-memory buffer or a file descriptor), which previously raised `AttributeError` or closed the caller's descriptor.

- **Read generic file-like streams in `convert_to_bytes()`**: streams such as `tempfile.TemporaryFile()` or file-like wrappers previously raised `ValueError("Invalid file-like object type")` from text-encoding detection and PDF page rendering. Previously unsupported file-like streams are now read from the start and rewound, including a `BufferedReader` whose `.name` is not a path (an in-memory buffer or a file descriptor), which previously raised `AttributeError` or closed the caller's descriptor.

cubic-dev-ai · 2026-10-02T04:17:07Z

+
+### Fixes
+
+- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0.


P3: detect_filetype() returns a FileType, not the input stream, so this says the wrong value is returned. Say the spooled file is left at position 0.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At CHANGELOG.md, line 5: <comment>`detect_filetype()` returns a `FileType`, not the input stream, so this says the wrong value is returned. Say the spooled file is left at position 0.</comment> <file context> @@ -1,3 +1,11 @@ + +### Fixes + +- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0. +- **Use `metadata_file_path` when a file's name is not a path**: a temporary file, including a rolled-over `SpooledTemporaryFile`, can report an integer file descriptor as its `.name`. File-type detection now ignores non-string names and falls back to `metadata_file_path` for the filename extension instead of raising `TypeError`. +- **Read any seekable binary stream in `convert_to_bytes()`**: streams such as `tempfile.TemporaryFile()` or file-like wrappers previously raised `ValueError("Invalid file-like object type")` from text-encoding detection and PDF page rendering. Any object with `read()` and `seek()` is now read from the start and rewound, including a `BufferedReader` whose `.name` is not a path (an in-memory buffer or a file descriptor), which previously raised `AttributeError` or closed the caller's descriptor. </file context>

Suggested change

- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0.

- **Avoid copying spooled uploads into memory**: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is left at read position 0.

CyMule added 4 commits July 22, 2026 22:49

fix(partition): avoid copying spooled uploads

25a6aa3

chore(release): add 0.25.2 changelog entry

84471e7

chore(partition): merge main into spooled upload optimization

a78b748

CyMule changed the title ~~fix(partition): avoid copying spooled uploads~~ fix(partition): read spooled uploads in place instead of copying them Oct 1, 2026

This was referenced Oct 1, 2026

perf(api): detect MIME types without copying the upload Unstructured-IO/unstructured-api#579

Open

fix(gzip): stop holding decompressed uploads in memory Unstructured-IO/unstructured-api#581

Open

CyMule marked this pull request as ready for review October 1, 2026 17:15

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread unstructured/partition/common/common.py

fix(filetype): preserve path-like stream filename hints

f2b4cf7

cubic-dev-ai Bot reviewed Oct 1, 2026

View reviewed changes

CyMule added 2 commits October 1, 2026 22:32

chore: merge main and retarget release to 0.27.12

8851293

fix(partition): read nameless buffered readers in convert_to_bytes

d145541

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

CyMule mentioned this pull request Oct 2, 2026

perf(pdf): bound OCR upload and rendered-page memory (0.27.13) #4516

Open

refactor(partition): rename spool helper to rewind_if_spooled and reu…

f898375

…se it

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

Comment thread unstructured/partition/common/common.py

CyMule marked this pull request as draft October 2, 2026 03:33

CyMule marked this pull request as ready for review October 2, 2026 04:12

cubic-dev-ai Bot reviewed Oct 2, 2026

View reviewed changes

This was referenced Oct 2, 2026

perf: Free decode buffers sooner when restoring original elements #4524

Draft

perf: Free image buffers sooner during figure extraction #4523

Draft

CyMule added 2 commits October 5, 2026 10:35

chore: Resolve merge conflicts with main

c2288b9

chore(none): Merge main and resolve PR conflicts

8f7d3c1

Navigation Menu

Repository navigation

fix(partition): read spooled uploads in place instead of copying them - #4419

fix(partition): read spooled uploads in place instead of copying them#4419
CyMule wants to merge 10 commits into
mainfrom
fix/avoid-spooled-upload-copies

CyMule commented Jul 23, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		### Fixes

		- Avoid copying spooled uploads into memory: `detect_filetype()` and the DOCX, PPTX, and shared partitioning paths now read `SpooledTemporaryFile` inputs in place instead of copying their complete contents into `BytesIO`. Large uploads remain disk-backed, avoiding a document-sized heap allocation without changing the partition API. A spooled file passed to `detect_filetype()` is still returned at read position 0.

Conversation

CyMule commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Release

Impact

Validation

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 2, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

CyMule commented Jul 23, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading