fix: stop copying a chunk's orig_elements twice on every serialization

[paulkarayan]

What & why

Problem: Anyone chunking a document was paying to photocopy the whole thing twice on every serialization, then bin both copies. ElementMetadata.to_dict() deep-copies every metadata field and then replaces coordinates, data_source, orig_elements and key_value_pairs with their serialized form, so the copies of those four are built and thrown away. Separately, _fix_metadata_field_precision() copies every element in order to round coordinates and detection_class_prob, which most elements do not have. On a chunk, orig_elements holds every source element of that chunk, so a single to_dict() duplicated the document twice over. In a profiled local pipeline over 45,000 elements, copy.deepcopy and its helpers accounted for roughly 40% of total run time.

There is a correctness consequence too. Element.id mints a uuid on first access and caches it on that element. Because the copies were the objects that got serialized, they took the freshly minted ids with them and the originals stayed unset, so serializing one chunk twice reported different element_id values for the same source elements each time.

Change: In to_dict(), deep-copy only the fields that are not separately serialized and rebuild the dict in its original key order, so the four re-serialized fields keep their position. In _fix_metadata_field_precision(), return the element untouched when it has neither coordinates nor detection_class_prob; mint the element's id before the copy that remains; and make that copy shallow, rebuilding CoordinatesMetadata instead of rounding the caller's points in place, so orig_elements is not duplicated and its nested ids stay stable.

Blast radius: 3/5 -- two functions on the shared serialization path that every caller of to_dict(), elements_to_json(), elements_to_dicts() and orig_elements inherits; small, self-contained, and revert-safe.

Linked ticket

none

Impact

Library users: chunk-heavy serialization gets materially faster, and repeated serialization of the same element now reports stable element_id values for its orig_elements. Measured on this branch, one probe run in both states, interleaved in a single process, minimum of N:

case	before	after	speedup
to_dict, no orig_elements	17.2 ms	18.3 ms	0.94x
to_dict, 10 orig_elements	323.1 ms	74.2 ms	4.36x
to_dict, 40 orig_elements	399.1 ms	62.4 ms	6.40x
elements_to_json, no orig_elements	77.9 ms	22.2 ms	3.51x
elements_to_json, 10 orig_elements	440.6 ms	75.3 ms	5.85x
elements_to_json, 40 orig_elements	666.1 ms	63.0 ms	10.57x

The first row is the one that did not improve and reads slightly worse: it pays four extra dict pops and has no orig_elements to skip. The machine was under heavy load during timing, so treat the magnitudes as approximate and that row as indistinguishable from noise.

Wire contract / clients: the serialized dict is unchanged in structure and in every value except element_id for elements that had none assigned, which was previously regenerated on each call. This reaches elements_to_json() and elements_to_ndjson() as well as the ids inside orig_elements. A caller that recorded those ids and expected a later serialization to produce the same ones was already getting different values every time; it now gets the same ones. This holds for an element that carries orig_elements together with its own coordinates or detection_class_prob as well, because the precision fix no longer deep-copies the orig_elements subtree. Elements given an explicit element_id, or one assigned by id_to_hash(), were never affected either way.

Metadata key order is preserved. An earlier revision of this branch popped the four separately-serialized fields before the copy and appended them afterwards, which moved coordinates, data_source, orig_elements and key_value_pairs to the end of the dict. elements_to_json(), elements_to_ndjson() and the base64 orig_elements all serialize with sort_keys=True so their bytes never changed, but Element.to_dict() and elements_to_dicts() are order-sensitive for a caller that writes them unsorted, and unstructured-ingest's chunker does exactly that. That turned test_ingest_src red on a fixture diff. The current revision rebuilds the dict in the original order and a test pins it.

Shared callers that inherit this: ElementMetadata.to_dict() is reached from Element.to_dict(), and therefore from elements_to_dicts() (and its convert_to_isd / convert_to_dict aliases), elements_to_json(), elements_to_base64_gzipped_json() and elements_to_ndjson(). _fix_metadata_field_precision() is called by elements_to_base64_gzipped_json() (base.py:256), elements_to_json() (base.py:453) and elements_to_ndjson() (base.py:475).

Risk / rollback

Low. Two functions, no signature or schema change, revert the commit to back it out. The one deliberate behavior change is the element_id stability described above.

How it was verified

Ran locally on Python 3.13 against this branch. Each of the three new tests was run first against the unfixed sources restored from HEAD with the tests in place, to confirm it fails for the reason claimed, and then against the fix.

Suites run: test_unstructured/chunking, test_unstructured/documents and test_unstructured/staging give 796 passed and 25 skipped on the current head, with one collection error in test_unstructured/staging/test_huggingface.py for a missing transformers, which is an uninstalled optional dependency rather than a failure. make check-ruff is clean. The wider test_unstructured tree passes apart from the partition and metrics trees, which need unstructured_inference, and cleaners/test_translate.py plus the benchmark test, which fail on missing sentencepiece and pytest-benchmark in my environment and fail the same way without this change.

Note on the environment, because it bit me: without the csv, docx, tsv and xlsx extras installed, test_unstructured/staging/test_base.py and test_unstructured/chunking/test_basic.py do not collect at all, and the run reports a confident 564 passed while silently skipping the file the precision tests live in.

Reviewed by fable and GPT-5.5 Pro before this leaves draft. Both independently found that the first commit left ids unstable for any element carrying coordinates or detection_class_prob, which is every hi_res-partitioned element, so its stability claim was false for the common case. My own test could not see it, because I built the fixture from bare Text elements with no coordinates. Fixed, with both variants now covered by a parametrized regression test. fable separately caught that the CHANGELOG.md heading has to carry the -dev0 suffix to match __version__ or scripts/version-sync.sh fails make check: the precedent is commit 4fe4097, whose heading is ## 0.27.5-dev0, and the release commit is what strips the suffix from both.

Not verified: scripts/version-sync.sh -c could not run locally, since it needs GNU sed 4.3 and macOS ships BSD sed, so CI is the first real check that the heading and __version__ agree. I also have not measured this on a GPU or OCR-heavy end-to-end partition, where model inference dominates and this saving is proportionally much smaller.

Proof

Repro. Profiled a 201-document, 45,000-element local pipeline with the real chunker under cProfile. copy.deepcopy was 10.783 s exclusive over 180,000 calls in a 40.018 s run, and with _deepcopy_list, _deepcopy_dict, _keep_alive and _deepcopy_atomic the deepcopy machinery totalled about 16.6 s. Reduced to a standalone reproduction of the id half:

$ python -c "from unstructured.documents.elements import Text, ElementMetadata; \
    md = ElementMetadata(orig_elements=[Text('hello world')]); e = Text('chunk', metadata=md); \
    print(e.to_dict() == e.to_dict())"
False

Decoding the base64 payload on three consecutive calls gave three different element_id values for the same nested element: 2ba29618-..., 4e806db8-..., 7070f009-....

Failing tests, run against the unfixed sources with the new tests in place.

$ python -m pytest test_unstructured/documents/test_elements.py -k "does_not_deep_copy or same_way_on_every_call" test_unstructured/staging/test_base.py -k "does_not_deep_copy or same_way_on_every_call or leaves_elements_with_nothing_to_round" -q
FAILED test_unstructured/documents/test_elements.py::DescribeElementMetadata::and_it_does_not_deep_copy_the_sub_objects_it_reserializes
FAILED test_unstructured/documents/test_elements.py::DescribeElementMetadata::and_it_serializes_orig_elements_the_same_way_on_every_call
FAILED test_unstructured/staging/test_base.py::test_fix_metadata_field_precision_leaves_elements_with_nothing_to_round_alone
3 failed, 145 deselected in 1.71s

After the fix, same tests, same command:

2 passed, 64 deselected in 2.41s     (documents)
1 passed, 81 deselected in 1.31s     (staging)

Second round, after review. With the id-mint line removed from staging/base.py:

$ python -m pytest test_unstructured/documents/test_elements.py -k "precision_still_has_to_be_rounded" -q
FAILED ...and_that_holds_for_elements_whose_precision_still_has_to_be_rounded[coordinates]
FAILED ...and_that_holds_for_elements_whose_precision_still_has_to_be_rounded[detection_class_prob]
2 failed, 66 deselected in 2.24s

With it restored: 2 passed, 66 deselected in 1.33s. All three element kinds checked directly:

with coordinates, two to_dict calls equal: True
with detection_class_prob, two to_dict calls equal: True
no coordinates, two to_dict calls equal: True

Suites for the touched areas:

$ python -m pytest test_unstructured/chunking test_unstructured/documents test_unstructured/staging -q
673 passed, 25 skipped in 13.18s

The differential probe, one command run in both states. This is the table under Impact; the row that did not move is the point of showing all six.

case                       baseline        fixed   speedup
to_dict, no origs           17.2 ms      18.3 ms     0.94x
to_dict, 10 origs          323.1 ms      74.2 ms     4.36x
to_dict, 40 origs          399.1 ms      62.4 ms     6.40x
to_json, no origs           77.9 ms      22.2 ms     3.51x
to_json, 10 origs          440.6 ms      75.3 ms     5.85x
to_json, 40 origs          666.1 ms      63.0 ms    10.57x

Still shaky. The timings were taken under load average 55, so their direction and rough scale are solid and the precise multiples are not. The 0.94x row cannot be separated from noise at that load.

Third round, after review. Three more tests, each run against the unfixed sources with the test in place.

Key order, test_unstructured/documents/test_elements.py:

FAILED ...and_it_emits_the_separately_serialized_fields_in_their_declared_position
E       At index 0 diff: 'filetype' != 'coordinates'

Nested id stability under the precision fix, test_unstructured/staging/test_base.py, both variants failing at the old head with differing uuids across two elements_to_json() calls:

FAILED ...test_elements_to_json_keeps_orig_elements_ids_stable_for_an_element_it_rounds[coordinates]
FAILED ...test_elements_to_json_keeps_orig_elements_ids_stable_for_an_element_it_rounds[detection_class_prob]

All three pass after the fix.

The third new test, test_fix_metadata_field_precision_rounds_a_copy_and_leaves_the_callers_element_unrounded, passes at both heads, because the old deepcopy already protected the caller. A test that cannot fail is worth nothing, so it was mutation-checked instead: with the shallow copy plus in-place rounding of points it fails on (1.23, 2.35) != (1.23456, 2.34567).

Not re-measured. The timing table above was taken before the key-order rebuild was added. That rebuild is one extra dict comprehension over the same fields, measured at 1.47 ms against 1.42 ms on a 40-element chunk, so the table's magnitudes still stand, but it is not a fresh run.

Not run. The ingest shell fixtures (test-ingest-src.sh) were not run locally. The key-order claim rests on a direct to_dict() probe against origin/main across three metadata shapes, not on the CI fixture diff, so CI is the first real check of it.

Dependencies / merge order

none

[paulkarayan]

Repro-first proof

Bugfix — ElementMetadata.to_dict and _fix_metadata_field_precision each deep-copied a chunk's whole orig_elements list and discarded the copy

Reproduced the broken state

• Environment: local
• How: Profiled a 201-document / 45,000-element local ingest pipeline (real Chunker.run on
  unstructured 0.27.5) under cProfile, then read the source at the top of the profile.
  Reproduced standalone with:
  python -c "from unstructured.documents.elements import Text, ElementMetadata; md = ElementMetadata(orig_elements=[Text('hello world')]); e = Text('chunk', metadata=md); print(e.to_dict() == e.to_dict())"
• Observed: copy.deepcopy was 10.783 s exclusive over 180,000 calls in a 40.018 s run, and the
  deepcopy machinery (_deepcopy_list, _deepcopy_dict, _keep_alive, _deepcopy_atomic)
  totalled about 16.6 s, roughly 40% of the run. Reading the source: to_dict() deep-copies
  every field and then overwrites coordinates, data_source, orig_elements and
  key_value_pairs with their serialized form, and _fix_metadata_field_precision() copies
  every element again to round two fields most elements do not carry.
  Side effect: the standalone repro above prints False. Element.id mints a uuid on first
  access and caches it, so each call materialized ids on the throwaway copies and reported
  different element_ids for the same source elements.
• Evidence: /Users/pk/orca/workspaces/wonderland-experimental/rust-and-clojure-ports/experiments/ingest-processes/RESULTS.md

Failing test (red)

• test_unstructured/documents/test_elements.py::DescribeElementMetadata::and_it_serializes_orig_elements_the_same_way_on_every_call (unit) — committed
• transcribed by the author (not captured by a runner)

$ .venv-poc/bin/python -m pytest test_unstructured/documents/test_elements.py -k "does_not_deep_copy or same_way_on_every_call" test_unstructured/staging/test_base.py -k "does_not_deep_copy or same_way_on_every_call or leaves_elements_with_nothing_to_round" -q
FAILED test_unstructured/documents/test_elements.py::DescribeElementMetadata::and_it_does_not_deep_copy_the_sub_objects_it_reserializes
FAILED test_unstructured/documents/test_elements.py::DescribeElementMetadata::and_it_serializes_orig_elements_the_same_way_on_every_call
FAILED test_unstructured/staging/test_base.py::test_fix_metadata_field_precision_leaves_elements_with_nothing_to_round_alone
3 failed, 145 deselected in 1.71s
(run against the unfixed sources, restored from HEAD, with the new tests in place)

Fix

• Drop the separately-serialized fields before the copy in to_dict(); skip the element copy in _fix_metadata_field_precision() when there is nothing to round
• Files: unstructured/documents/elements.py, unstructured/staging/base.py

Proof it's resolved

• Test green: yes
• Environment: local
• Evidence: `Same three tests, same command, against the fixed sources:
  2 passed, 64 deselected (elements) + 1 passed, 81 deselected (staging)
  Full suites for the touched areas:
  test_unstructured/chunking + documents + staging -> 671 passed, 25 skipped
  test_unstructured (minus partition/metrics, heavy deps) -> 1492 passed, 25 skipped,
  5 failed, 1 error. All 6 are environment gaps present before this change:
  cleaners/test_translate.py needs sentencepiece, the benchmark test needs
  pytest-benchmark.

One probe, both states, interleaved in a single process, min of N (bench2.py):
case baseline fixed speedup
to_dict, no origs 17.2 ms 18.3 ms 0.94x
to_dict, 10 origs 323.1 ms 74.2 ms 4.36x
to_dict, 40 origs 399.1 ms 62.4 ms 6.40x
to_json, no origs 77.9 ms 22.2 ms 3.51x
to_json, 10 origs 440.6 ms 75.3 ms 5.85x
to_json, 40 origs 666.1 ms 63.0 ms 10.57x
The no-origs to_dict row is the one that did not improve and reads slightly worse; it
pays four extra dict pops and gains nothing, since there is no orig_elements to skip.`

---

Auto-generated from this branch's .proof.toml (repro-first proof gate). Advisory.

[paulkarayan]

Strong review done (2026-09-05, fable + gpt-pro) -- verdict: two blocking findings, both fixed in dbc99bd before this left draft. Advisory report kept locally; re-run only on purpose (new commits that change the risk surface).

[cubic-dev-ai]

All reported issues were addressed across 6 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[tabossert]

The code changes look correct: to_dict() now deep-copies only the non-separately-serialized fields while keeping declared key order, and _fix_metadata_field_precision() skips the copy when nothing needs rounding and mints the id before copying, so element_id is stable across serializations. Approving on the code, but this cannot merge until the conflict below is resolved.

Blocking

1. Merge conflict with main in CHANGELOG.md and unstructured/__version__.py (the only two conflicting files; git merge-tree against current main). This PR has 0.27.11-dev0, but main is at __version__ = "0.27.16" with 0.27.11 through 0.27.16 already released. Keeping this side when resolving would move __version__ backwards and file the entry under an already-shipped 0.27.11. The changelog enforcer and make check-version (scripts/version-sync.sh) only check that the files changed and that no release is duplicated, so CI would not catch a lower version. Fix: merge or rebase onto main, move the entry into a new ## 0.27.17-dev0 section above ## 0.27.16, set __version__ = "0.27.17-dev0", and run make check-version before pushing. The code files need no changes (none of them have moved on main since the merge-base).

Non-blocking

1. unstructured/staging/base.py:492: the docstring says "the caller's element is not modified", but _ = element.id at l.506 assigns and caches a uuid on the caller's element when it has none. That mutation is intended (the CHANGELOG says ids are now minted on the caller's element, and the inline comment above l.506 explains why), but the docstring contradicts both. Someone trusting it could drop _ = element.id as a no-op and bring back unstable ids for elements with coordinates or detection_class_prob. Suggest: "the caller's element is never rounded, but if it has no id, a uuid is minted and cached on it so element_id stays stable across calls."

2. test_unstructured/documents/test_elements.py:458: the docstring of and_it_emits_the_separately_serialized_fields_in_their_declared_position names key_value_pairs as one of the fields whose position must be kept, but the fixture never sets it and the expected key list does not include it. It is the only one of the four SEPARATELY_SERIALIZED_FIELD_NAMES the test does not pin. The placeholder rebuild treats all four the same way today, but if key_value_pairs is ever special-cased, a key-position change (and changed bytes from consumers that do not sort keys) would go unnoticed. Suggest adding key_value_pairs=[FormKeyValuePair(...)] (with a Text custom_element) to the fixture and putting "key_value_pairs" between "filetype" and "languages" in the expected list, or drop it from the docstring.

Questions

1. unstructured/staging/base.py:488: was rounding the serialized dicts considered instead of copying Elements? All three callers (l.257, 454, 477) run this helper and then elements_to_dicts(), and all dump with sort_keys=True. coordinates.to_dict() emits points unchanged and detection_class_prob passes through as-is. Both fixes this helper has needed (nested orig_elements ids, and the mint-before-copy ordering for _ = element.id) come from copying an Element just to round two values. Rounding d["metadata"]["coordinates"]["points"] after elements_to_dicts() (1 decimal when system == "PixelSpace", which matches the current isinstance since there are no subclasses, else 2) and d["metadata"]["detection_class_prob"] (5 decimals) would copy nothing, mint ids on the caller's element naturally, drop the new CoordinatesMetadata import, and should produce identical bytes. The two tests at test_unstructured/staging/test_base.py:1016 and :1031 that call the private helper would need rewriting. If you rejected this, a line in the PR description saying why would help.

[paulkarayan]

@tabossert late answer, and this merged, so here is the record.

Blocking, the CHANGELOG and version conflict: resolved. Rebased onto main, the entry sits in a new ## 0.27.17 section above ## 0.27.16, and __version__ is 0.27.17. One difference from what you asked for: it went in as a release version rather than 0.27.17-dev0.

Non-blocking 1, the contradictory docstring: not fixed. _fix_metadata_field_precision still says "the caller's element is not modified" while _ = element.id mints and caches a uuid on it. Your wording is the right one, and the risk you named is real: someone reading that line can delete the id access as a no-op and bring back unstable ids for every element with coordinates or detection_class_prob.

Non-blocking 2, key_value_pairs in the key-position test: not fixed. The docstring still names it as one of the four fields whose position must hold, and the fixture still does not set it, so the expected key list is coordinates, data_source, filetype, languages, orig_elements, page_number. It is still the only one of SEPARATELY_SERIALIZED_FIELD_NAMES the test leaves unpinned.

Question 1, rounding the serialized dicts instead of copying Elements: no, that was not tried, and the PR body carries no line saying why. Reading it now your version looks better on the merits, for the reason you give: both corrections this helper has needed (nested orig_elements ids, and minting the id before the copy) exist only because it copies an Element to round two numbers. Rounding d["metadata"]["coordinates"]["points"] and d["metadata"]["detection_class_prob"] after elements_to_dicts() would copy nothing and drop the new CoordinatesMetadata import. The cost is rewriting the two tests that call the private helper. I have not measured that it produces identical bytes, so treat it as an open suggestion rather than a decision.