[tanishqbaweja]

Summary

Since #4391, partition_json() and partition_ndjson() turn arbitrary JSON (anything that isn't serialized Unstructured elements) into Text elements that hold the pretty-printed JSON. That text comes from pretty_json_text() in unstructured/partition/common/json_partitioning.py, which called json.dumps(value, indent=2, sort_keys=True). ensure_ascii defaults to True, so every non-ASCII character reached Element.text as a six-character \uXXXX escape, and a character outside the Basic Multilingual Plane, such as an emoji, as two of them. partition() on a .json or .ndjson file goes through the same code.

import io
import json

from unstructured.partition.json import partition_json

data = json.dumps([{"title": "東京の天気", "city": "São Paulo"}], ensure_ascii=False).encode()
print(partition_json(file=io.BytesIO(data))[0].text)

# main
{
  "city": "S\u00e3o Paulo",
  "title": "\u6771\u4eac\u306e\u5929\u6c17"
}

# this PR
{
  "city": "São Paulo",
  "title": "東京の天気"
}

The repo's own example-docs/test_evaluate_files/gold_standard_table_structure/IRS-2023-Form-1095-A.pdf.json, an arbitrary payload for partition_json(), shows it too: on main, 19 of its 81 elements hold escapes, such as "3 Policy issuer\u2019s name", and with this PR they read "3 Policy issuer’s name".

Japanese, Chinese or Russian JSON becomes runs of escapes in the text that gets embedded, searched and passed to an LLM, and accented Latin words are split by escapes. The text also gets longer and makes more chunks. Continuing the example above:

from unstructured.chunking.basic import chunk_elements

records = [
    {"title": "東京の天気予報", "body": "明日は晴れのち曇り、最高気温は二十五度の予想です。"},
    {"titre": "Fête à Montréal", "texte": "Les élèves étaient très enthousiastes à l'idée."},
    {"заголовок": "Погода в Москве", "текст": "Завтра ожидается дождь и сильный ветер."},
]
elements = partition_json(text=json.dumps(records, ensure_ascii=False))
print([len(e.text) for e in elements])  # main: [223, 139, 390]   this PR: [63, 94, 90]
print(len(chunk_elements(elements, max_characters=120)))  # main: 9   this PR: 3
print(len(chunk_elements(elements)))  # default max_characters=500; main: 2   this PR: 1

On main the text holds escapes whether the file stores the UTF-8 characters or escapes. With this PR it holds the characters in both cases, so a file that Python's json.dump() saved with its default ensure_ascii=True gives the same text as one saved with ensure_ascii=False. #4483 keeps valid UTF-8 input exact for both partitioners; for arbitrary JSON, pretty_json_text() then escaped the characters again.

pretty_json_text() now calls json.dumps() with ensure_ascii=False. The text is still valid JSON, and json.loads(element.text) returns the same value as before.

A bare ensure_ascii=False would break one input that works on main. A JSON escape can hold one half of a UTF-16 surrogate pair on its own ("\ud800"), and json.loads() turns it into a lone surrogate, which UTF-8 cannot encode. Written out as it is, that character makes partition_json() and partition_ndjson() raise UnicodeEncodeError ("surrogates not allowed") in Element.id_to_hash(), which hashes the UTF-8 bytes of the text. So pretty_json_text() writes any surrogate back as the escape json.dumps() writes by default. Such a payload still partitions, and its other non-ASCII characters are kept.

DEL (U+007F) is ASCII, but json.dumps() escapes it by default (\u007f), so pretty_json_text() keeps that escape too. For content that is all ASCII, the text is exactly what main writes.

Serialized elements that are rehydrated are not affected: their text comes from the text field, not from pretty_json_text(). A payload with no non-ASCII characters (after its escapes are decoded) gives the same text, element ids and metadata as before. No fixtures or expected outputs change. The arbitrary-JSON example docs that tests partition (arbitrary-records.json, arbitrary-records.ndjson, not-unstructured-payload.json and single-line-object.json) are pure ASCII. The gold_standard_table_structure files, such as the IRS form above, are only read by the metrics code, with json.load(). No file under test_unstructured_ingest/expected-structured-output* comes from a .json or .ndjson input.

NDJSON line breaks

partition_ndjson(), the NDJSON check in detect_filetype() and unstructured.file_utils.ndjson.loads() split their input with str.splitlines(). Besides LF, CRLF and CR, that also breaks at U+0085 (NEL), U+2028 and U+2029, which JSON allows unescaped inside a string; json.dumps(..., ensure_ascii=False) and JavaScript's JSON.stringify() write them that way. A record holding one is cut in two. On main, partition_ndjson() raises ValueError("Not a valid ndjson") for such a file, and a .json file of two such records is detected as JSON instead of NDJSON.

With the text change above, this also broke a round trip (thanks for the review): element text now holds these characters as themselves, so NDJSON written from the element dicts with json.dumps(..., ensure_ascii=False) holds them raw, and reading it back raised the same error. On main the round trip worked only because the text kept the escape.

The second commit adds split_lines() to unstructured/file_utils/ndjson.py, and all three callers use it. It splits at the other line breaks str.splitlines() knows (CRLF, LF, CR, and the control characters VT, FF, FS, GS and RS) but not at U+0085, U+2028 or U+2029. JSON allows none of those control characters unescaped inside a string, so splitting at them can't cut a valid record. For text without U+0085, U+2028 or U+2029 it returns exactly what str.splitlines() returns (checked on 84,303 random strings built from these characters), so CR-only input and the other line breaks are read as before.

Validation

New tests:

• test_unstructured/partition/common/test_json_partitioning.py, for pretty_json_text():
  • it_writes_non_ASCII_characters_as_themselves_rather_than_as_escapes: the exact text for Portuguese, Japanese and an emoji.
  • but_it_keeps_a_lone_surrogate_escaped_so_the_text_can_be_encoded_as_UTF_8: the exact text, with the surrogate escaped and São Paulo kept. The text encodes as UTF-8 and json.loads() returns the original value.
  • and_it_still_escapes_control_characters_and_DEL: a tab and a newline still come out as \t and \n, and DEL as the escape json.dumps() writes by default.
• test_unstructured/partition/test_json.py, in a new section after the UTF-8 detection tests:
  • it_keeps_non_ASCII_characters_in_the_text_of_an_arbitrary_payload[False|True]: a file= payload dumped with ensure_ascii=False or with ensure_ascii=True gives the same exact text.
  • and_it_keeps_them_when_reading_an_arbitrary_payload_from_a_filename
  • it_partitions_an_arbitrary_payload_holding_a_lone_surrogate_escape: no error, and json.loads(element.text) returns the original value.
• test_unstructured/partition/test_ndjson.py, in the same place: the same three checks for NDJSON lines.

# main source + the new tests
FAILED common/test_json_partitioning.py::it_writes_non_ASCII_characters_as_themselves_rather_than_as_escapes
FAILED common/test_json_partitioning.py::but_it_keeps_a_lone_surrogate_escaped_so_the_text_can_be_encoded_as_UTF_8
FAILED test_json.py::it_keeps_non_ASCII_characters_in_the_text_of_an_arbitrary_payload[False]
FAILED test_json.py::it_keeps_non_ASCII_characters_in_the_text_of_an_arbitrary_payload[True]
FAILED test_json.py::and_it_keeps_them_when_reading_an_arbitrary_payload_from_a_filename
FAILED test_ndjson.py::it_keeps_non_ASCII_characters_in_the_text_of_arbitrary_lines[False]
FAILED test_ndjson.py::it_keeps_non_ASCII_characters_in_the_text_of_arbitrary_lines[True]
FAILED test_ndjson.py::and_it_keeps_them_when_reading_arbitrary_lines_from_a_filename
8 failed, 3 passed

# this PR
11 passed

The 3 tests that pass on main (the control characters and DEL, and the two lone-surrogate partition tests) guard the escaping this PR keeps. With a bare ensure_ascii=False in place of this change, but_it_keeps_a_lone_surrogate_escaped_so_the_text_can_be_encoded_as_UTF_8 and and_it_still_escapes_control_characters_and_DEL fail, and both lone-surrogate partition tests fail with the UnicodeEncodeError above.

Other runs on Windows (Python 3.12):

• test_json.py, test_ndjson.py and common/test_json_partitioning.py: 11 failed, 190 passed on main with the new tests, and 3 failed, 198 passed with this PR. The 3 that fail on both come from this machine. Two tests expect "nested too deeply" from pretty-printing a 5,000-deep array but get "Not a valid json" (or "ndjson"), because json.loads() already hits its recursion limit on this Windows build. it_rehydrates_on_a_detect_then_partition_sequence_over_the_same_file_handle needs libmagic. Calling pretty_json_text() on a 5,000-deep list built in Python raises ValueError("JSON value is nested too deeply to partition") on main and with this PR.
• test_unstructured/partition/test_auto.py -k json: 13 passed, 1 xfailed, on main and with this PR.
• pretty_json_text(chr(n)) for every code point: each one from U+0020 up, other than ", \, DEL and the 2,048 surrogates, comes out as itself (1,112,029 in all). The characters below U+0020, ", \, DEL and every surrogate (2,083) come out exactly as json.dumps() writes them by default, so every ASCII character gives the same text as on main. Each result encodes as UTF-8, and json.loads() returns the character.
• example-docs/arbitrary-records.json, arbitrary-records.ndjson, not-unstructured-payload.json and single-line-object.json give identical element dicts (text, ids and metadata) on main and with this PR.

ruff check . and ruff format --check . pass (ruff 0.15.10, the version in uv.lock).

Second commit (NDJSON line breaks):

• test_unstructured/partition/test_ndjson.py:
  • it_keeps_a_line_whole_when_a_string_in_it_holds_a_Unicode_line_separator, for U+0085, U+2028 and U+2029.
  • and_its_elements_round_trip_through_NDJSON_written_without_escapes, for the same three: partition a record holding the separator, write the element dicts with json.dumps(..., ensure_ascii=False), and partition that NDJSON again.
  • it_still_splits_lines_on_a_lone_carriage_return: passes before and after; it pins CR-only input.
• test_unstructured/file_utils/test_filetype.py::and_it_classifies_lines_whose_strings_hold_a_Unicode_line_separator_as_NDJSON, for the same three: a .json file of two such records is NDJSON.

On the first commit's code these 10 tests give 9 failed, 1 passed (the CR-only one), and with the second commit 10 passed. On main (910d9f41e) the keep-whole and detection cases fail the same way, and the round trip passes, since main kept the escape.

test_ndjson.py, test_json.py, common/test_json_partitioning.py, file_utils/test_filetype.py and test_unstructured/staging: 36 failed, 657 passed before the second commit, and 27 failed, 666 passed after it. The only difference is the 9 new tests. The 27 fail on this machine either way: libmagic is not installed here, and the two "too deep" tests hit the recursion limit as described above. test_auto.py -k "json or ndjson": 13 passed, 1 xfailed. ruff 0.15.10 check and format --check pass on the changed files.

Release note (optional)

The Text elements that partition_json(), partition_ndjson() and partition() make from arbitrary JSON and NDJSON now hold non-ASCII characters as themselves instead of \uXXXX escapes, whether the file stores the characters or escapes. The text is still valid JSON, and json.loads() of it returns the same value as before. For elements whose content has non-ASCII characters, the text is shorter, so chunk boundaries can move, and the deterministic element ids, which are hashed from the text, change. Elements whose content is all ASCII, and rehydrated serialized elements, are unchanged. A lone surrogate (an escape such as \ud800 that is not half of a pair) still comes out as an escape. NDJSON whose strings hold U+0085, U+2028 or U+2029 unescaped is now read instead of raising ValueError("Not a valid ndjson"), and a .json file of such records is detected as NDJSON.

Compatibility and reindexing: re-partitioning the same JSON or NDJSON file gives different text, and so different element ids, for every arbitrary-JSON element whose content has non-ASCII characters. With a chunking strategy, chunk boundaries can move, which also changes the ids of later chunks. An index or cache keyed by element id, such as a vector store that upserts by id, should re-ingest those files; otherwise it keeps the old elements next to the new ones. Elements that are all ASCII and rehydrated serialized elements keep their text and ids. How to release and announce this is your call.

Checklist

• The PR title is a user-facing release-note entry with a conventional commit prefix.
• Apply a category label (enhancement, bug, or an appropriate existing label), or ask a maintainer to apply it. This is a bug fix; could a maintainer please apply the bug label? I can't set labels on this repository.
• For breaking changes, use ! in the title and explain impact and migration above. The text and ids of arbitrary-JSON elements with non-ASCII content change, as the release note says; I've left the ! decision to you.

[cragwolfe]

P2 — Preserve NDJSON round trips for Unicode line separators. pretty_json_text() now emits U+0085, U+2028, and U+2029 literally, but partition_ndjson() splits input with str.splitlines() (unstructured/partition/ndjson.py:95), which treats those characters as record boundaries even inside JSON strings. For example, partition an NDJSON record containing "a\u2028b", serialize the resulting element dictionary with json.dumps(..., ensure_ascii=False), then reimport it as NDJSON: the embedded separator splits the serialized element and raises ValueError("Not a valid ndjson"). The earlier formatter kept the inner escape, so this round trip avoided that split. Please split on the supported physical NDJSON delimiters and add round-trip coverage for these separators; preserve any intended CR-only input support.

The change also intentionally changes public text for non-ASCII arbitrary JSON and NDJSON. Reprocessing can change deterministic element IDs, and chunk boundaries and later IDs can change when chunking is enabled. Please document the compatibility and reindexing implications; a maintainer should decide the release and migration treatment. Rehydrated element text does not pass through this formatter.

Reviewed the actual-base diff and surrounding callers and tests with GPT Pro. No local tests or reproductions were run. This comment does not indicate CI or integration readiness.

(authored by codex)

[tanishqbaweja]

Thanks for catching the round trip. Fixed in 747e08e:

• NDJSON is now split only at line breaks a JSON string can't hold. A new split_lines() in file_utils/ndjson.py is used by partition_ndjson(), the NDJSON check in detect_filetype() and file_utils.ndjson.loads(). U+0085, U+2028 and U+2029 no longer end a record. CRLF, LF, CR and the control-character breaks split as before: on text without those three characters the result is identical to str.splitlines(), so CR-only input still works.
• Round-trip tests for all three separators: partition, write the element dicts with json.dumps(..., ensure_ascii=False), partition again. There are also tests for raw input and for .json detection, and a CR-only test that pins the existing behaviour.
• This also fixes the same split on main. There, NDJSON with these characters unescaped, as JSON.stringify() writes them, raises "Not a valid ndjson", and a .json file of such records is detected as JSON.
• The release note now covers the compatibility and reindexing side: different text and ids for non-ASCII arbitrary-JSON elements, chunk boundaries and later chunk ids that can move, and a re-ingest for id-keyed stores. The release and migration treatment is your call.