[linhongyu510]

Problem

partition_json() and partition_ndjson() read the source with a hard-coded
UTF-8, and neither accepts an encoding parameter:

if filename is not None:
    with open(filename, encoding="utf8") as f:      # hard-coded
        file_text = f.read()

elif file is not None:
    file_content = file.read()
    file_text = file_content if isinstance(file_content, str) else file_content.decode()  # UTF-8

Because both signatures end in **kwargs: Any, an encoding= passed by the
caller — directly, or forwarded by partition() — is swallowed silently. On
main (ee2b3a35):

>>> partition_json(filename="utf16.json", encoding="utf-16")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The same call succeeds for the sibling text formats, which is where the
inconsistency shows:

partitioner	takes encoding	reads via
partition_text	yes	read_txt_file()
partition_csv	yes	passed to the reader
partition_json	no	open(..., encoding="utf8")
partition_ndjson	no	open(..., encoding="utf8")

UTF-16 JSON is not exotic — it is what several Windows tools emit by default,
and json.dump to a file opened in a non-UTF-8 encoding produces it too.

Fix

Add encoding to both signatures and read through the existing
read_txt_file() helper, exactly as partition_text() does:

if filename is not None:
    _, file_text = read_txt_file(filename=filename, encoding=encoding)

elif file is not None:
    _, file_text = read_txt_file(file=file, encoding=encoding)
    file.seek(0)

read_txt_file() uses the given encoding when one is supplied and falls back to
detect_file_encoding() when it is not, so this also makes a non-UTF-8 document
readable without naming its encoding. The text= path is untouched, and
file.seek(0) is preserved so the detect-then-partition sequence over one handle
still works.

Source change is +11/-8 across the two files; no new helper, no behaviour change
for UTF-8 input.

Verification

$ pytest test_unstructured/partition/test_json.py test_unstructured/partition/test_ndjson.py
35 failed, 75 passed

The 35 failures are pre-existing on main in this environment — they come from
unstructured/nlp/tokenize.py failing to import an optional dependency
(ModuleNotFoundError: No module named 'installer'), unrelated to these files.
I compared the failure sets with git stash: identical before and after, and
the pass count goes 70 → 75, which is exactly the five tests added here.

Five regression tests, following the existing naming style in both files:

• explicit encoding="utf-16" for filename= (both partitioners)
• explicit encoding="utf-16" for file= (both partitioners)
• no encoding given, UTF-16 detected (json)

Rollback proof. Restoring just the two read blocks to the hard-coded form —
leaving the new parameter in place — fails exactly those five and nothing else:

FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
FAILED test_json.py::it_detects_the_encoding_of_a_non_utf8_file_when_none_is_specified
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
5 failed, 105 deselected

each with UnicodeDecodeError. ruff check on the four changed files is clean,
and a CHANGELOG entry is added under the current 0.27.8-dev0 section.

Notes

This is the parsing side. There are open PRs (#4397, #4398, #4447) addressing
non-UTF-8 crashes during file-type detection in file_utils/filetype.py; I
checked all currently open PRs and none touches partition/json.py or
partition/ndjson.py, so this does not overlap with them.

I kept the scope to honoring the parameter. I did not change what happens when a
document really is undecodable, nor touch the text= path.

[cragwolfe]

Requesting changes for four production blockers:

1. [P1] Preserve valid UTF-8 when no encoding is supplied (unstructured/partition/json.py:87,90, unstructured/partition/ndjson.py:84,87). The old paths decoded bytes strictly as UTF-8. The new paths call read_txt_file(..., encoding=None), which accepts a sufficiently confident charset_normalizer.detect() guess before attempting UTF-8. With the locked charset-normalizer==3.4.7, valid UTF-8 JSON containing a literal zero-width space can be selected as CP932 and still parse successfully, changing the element text (for example, R​AG becomes R窶帰G). This is silent content corruption. Please make explicit encodings authoritative and, when encoding is omitted, preserve valid UTF-8 before falling back to non-UTF-8 detection. Add assertions on exact Unicode content for JSON and NDJSON through both filenames and binary streams; the current ASCII assertions do not cover this regression.

2. [P1] Preserve the existing file-like contract (unstructured/partition/json.py:90-91, unstructured/partition/ndjson.py:87-88). The removed implementation accepted any readable object and preserved already-decoded str values. The new no-encoding path routes the live object through convert_to_bytes(), which only accepts a few concrete types, may reopen a stream by .name, and uses BytesIO.getvalue() rather than reading from the current cursor. Exact-head CI already demonstrates the regression: both test_unit (3.13) and test_dockerfile fail in test_formskeysvalues_reads_saves because partition_json(file=io.StringIO(...)) now raises ValueError: Invalid file-like object type. Read the supplied stream once from its current position, pass str through, and decode buffered bytes without reopening or taking ownership of the caller's stream. Cover text streams, generic seekable binary streams, temporary-file wrappers, and nonzero cursor positions for both formats.

3. [P1] Forward encoding through the public dispatcher (unstructured/partition/auto.py:266-269). partition() binds encoding as a named parameter and uses it for detection, but its JSON/NDJSON special case calls the selected partitioner with only filename, file, and **kwargs. Because the bound value is not in kwargs, partition(..., encoding=...) still discards the caller's requested codec before parsing, leaving the stated top-level behavior unfixed and potentially producing different text or error behavior from a direct partition_json()/partition_ndjson() call. Pass encoding=encoding explicitly and add dispatcher-level tests for both formats using bytes that have different meanings or validity under competing codecs; a BOM-bearing UTF-16 success alone can pass through autodetection while the argument is still dropped.

4. [P2] Keep the existing fourth positional argument compatible (unstructured/partition/json.py:36-42, unstructured/partition/ndjson.py:36-42). Before this change, metadata_last_modified was the fourth positional parameter. Inserting encoding ahead of it silently rebinds existing calls: binary input attempts to use the timestamp as a codec and raises LookupError, while text= input can succeed but lose the requested timestamp. Preserve the original parameter order and place the new parameter afterward (or otherwise make only the new parameter keyword-only), with regression coverage for four-positional-argument calls in both partitioners.

The branch also currently conflicts with main in CHANGELOG.md; that is operational follow-up after the content blockers above are addressed.

(authored by codex)

[linhongyu510]

Thanks — all four are addressed, and the branch is rebased onto current main so the CHANGELOG conflict is gone. Both entries are kept, with the text rewritten to describe what the code now does.

I reproduced each one before changing anything, against charset-normalizer==3.4.7 as you specified.

1. Valid UTF-8 is preserved when no encoding is supplied. Confirmed, and worth recording how it reproduces: my first attempt did not show corruption because json.dumps() defaults to ensure_ascii=True, which escapes the zero-width spaces and leaves the payload pure ASCII. With ensure_ascii=False the real bytes appear and detection picks CP932 at confidence 0.907 — A\u200bB\u200bC… decodes to A窶毅窶気窶汽… and still parses as JSON. main is unaffected because it decodes strictly as UTF-8, so this was a regression I introduced.

Decoding now tries UTF-8 first and only falls back to detection for bytes that are not valid UTF-8. An explicit encoding is used as given, so a genuine mismatch surfaces as an error rather than mojibake.

2. The file-like contract is restored. Reproduced exactly as CI showed — partition_json(file=io.StringIO(...)) raised ValueError: Invalid file-like object type. The stream is again read once from its current position, an already-decoded str passes through untouched, and the object is never reopened or taken ownership of. Covered for text streams, binary streams, TemporaryFile wrappers and a nonzero cursor, in both formats.

3. encoding is forwarded through the dispatcher. Confirmed: it is bound as a named parameter of partition(), so it never reached the partitioner through **kwargs. Now passed explicitly. The dispatcher tests use BOM-less UTF-16-LE precisely because autodetection cannot recover it — a BOM-bearing payload would pass even with the argument still dropped.

4. Positional order is preserved. metadata_last_modified is the fourth positional parameter again, with encoding after it; regression tests call both partitioners with four positional arguments and assert the timestamp lands.

Both partitioners now share one read_json_text() helper, so the two paths cannot drift apart again.

Verification

• 119 pass across test_json.py and test_ndjson.py. The 3 remaining failures are identical to main on this machine and unrelated to this change (…too_deep_to_pretty_print ×2 and it_rehydrates_on_a_detect_then_partition_sequence_over_the_same_file_handle); main shows 102 passed / the same 3 failed.
• Rollback counter-proof: reverting only the UTF-8-first decode fails exactly the 8 new preservation/stream tests; reverting only the dispatcher forwarding makes both BOM-less UTF-16-LE cases fail with Not a valid json / Not a valid ndjson. Each group of tests pins one of the fixes.
• ruff check and ruff format --check clean on all 7 changed files.

One note on test placement: the dispatcher tests live in test_auto.py alongside the other partition() routing tests, but I could not execute that file locally — it imports pdf2image/PIL at module scope and the PDF stack is not installable in this environment. I verified the same two scenarios through a standalone script instead (including the rollback check above), so CI will be the first full run of those two tests.

[linhongyu510]

@cragwolfe Requesting a re-review when you have a moment — all four P1s from your 09-15 review were addressed in the 09-16 push (head fb8f5bb7), and the PR has been sitting on the now-outdated changes_requested since.

Not re-litigating the detail here since it is in my 09-16 reply, just the state: CI on this head is green across the board, and there are no unresolved review threads left. The only thing holding mergeable_state at BLOCKED is the review itself.

If anything in the responses did not land the way you intended, I would rather hear that than have it sit — happy to keep iterating.

[linhongyu510]

Gentle follow-up: head fb8f5bb7 addresses all four review blockers. CI is green — 53 checks success, including test_unit on 3.11/3.12/3.13 and the test_unit_dependency_extras matrix. Awaiting re-review when you have a chance.

[cragwolfe]

Re-reviewed exact head fb8f5bb74cc3f3f3ec2f8bf660e48b62e2b9cdf8 against fresh main (0b7f806e0f6ef540df22fad5627756a72663eb11), with a complete production-context Oracle review using server-confirmed GPT-6 Pro. No actionable introduced source defects remain. The four September 15 blockers are addressed: valid UTF-8 is preserved before fallback detection, caller streams are read from their current position without reopening or taking ownership, explicit encodings reach both partitioners through partition(), and the fourth positional argument remains metadata_last_modified.

This approval covers the reviewed source change. The PR still has substantive conflicts in unstructured/partition/json.py and test_unstructured/partition/test_json.py; CHANGELOG.md merges cleanly. Resolve using main's JSON identity/metadata processing from #4504, then integrate the encoding parameter and reader without restoring the old processing decorators.

Main's #4550 now supplies a BOM-aware, NUL-aware shared decoding policy. Keep the new read-once/string-pass-through stream acquisition and authoritative explicit decoding; prefer that shared policy for omitted-encoding buffered bytes. Do not pass the live caller stream into the detector. Main still lacks the JSON/NDJSON input-encoding support and dispatcher forwarding, so it does not supersede the whole PR. If default decoding is consolidated, make the dispatcher tests assert forwarding deterministically: BOM-less UTF-16 may now succeed through detection even if forwarding is removed.

Exact-head CI has 53 completed checks: 52 success and one neutral, including all six required checks. They completed September 16 and do not validate today's main integration. Preserve both sets of regression tests and obtain fresh CI on the conflict-resolved head before merging. No local tests were executed for this review.

(authored by codex)

[cragwolfe]

Fresh Oracle Pro production re-review of exact head 240b72904eede76d206f23deb490e49a662cef63 against current main (0b7f806e0f6ef540df22fad5627756a72663eb11). One verified P2 compatibility regression remains: the new named encoding parameter is omitted from JSON's manual chunker dispatch. The inline comment gives the failure case and fix. This is limited to registered chunkers that accept encoding; built-in chunkers are unaffected. No other actionable source defect was established.

The integration has resolved the prior JSON/source-test conflicts, preserved #4504's JSON identity/metadata processing, and adopted #4550's shared BOM/NUL-aware detector on buffered bytes. Explicit byte decoding, stream acquisition/ownership, fourth-position metadata_last_modified, and dispatcher forwarding are intact. I fetched all current issue comments, reviews, inline comments and thread replies; no posted discussion newer than the previous approval was returned. The subsequent integration commits were reviewed afresh.

Coverage follow-up: the two dispatcher tests still assume BOM-less UTF-16 cannot autodetect. That premise no longer proves forwarding with the shared detector. Add a deterministic assertion on the received codec, or make fallback decoding fail in the explicit-encoding case. Current dispatcher forwarding is correct, so this is not a separate source defect.

Merge readiness remains unvalidated: CI, CodeQL, Security and Partition Benchmark are action_required, with zero jobs on this head. The three passing external checks do not supply unit/integration execution; Cubic explicitly skipped AI review for this merge commit. Historical September CI does not cover this integration. The current merge tree is clean. After the chunker fix, obtain workflow execution and review on the new exact head before merging.

Oracle was available as server-confirmed gpt-6-pro, with Pro verified before submission and while streaming. Its complete final answer was recovered from the same conversation after CLI capture detached. Findings were verified against source and tests as code; no local tests or source changes were made.

(authored by codex)

[cragwolfe]

[P2] Preserve encoding when dispatching registered JSON chunkers

Binding encoding here removes it from kwargs, but _apply_chunking() only reconstructs filename, file, text, metadata_last_modified and the remaining kwargs. Both calls therefore drop the supplied codec. On current main, an explicit encoding stayed in kwargs and reached a registered chunker that declared it. For example, a chunker with a required keyword-only encoding now raises TypeError for partition_json(text='{}', encoding='utf-8', chunking_strategy='custom'); an optional codec parameter silently uses its default. NDJSON's decorator still forwards named arguments.

Pass the bound codec into _apply_chunking() and its argument map from both the rehydration and arbitrary-JSON paths, while retaining #4504's manual finalization. Extend the existing complete-call-arguments capture test to declare and assert encoding for rehydrated, arbitrary and empty input. Built-in chunkers need no behavior change.

(authored by codex)

[linhongyu510]

Follow-up on the 10-14 P2 note (chunker dispatch / coverage follow-up), addressed on head c4cebcf3:

• P2 chunker dispatch: encoding is now forwarded through JSON's manual _apply_chunking in both the rehydration and arbitrary-JSON paths (unstructured/partition/json.py). Registered chunkers that declare encoding receive it; built-in chunkers are unaffected because they don't declare it and the dispatcher filters kwargs by signature as before.
• Coverage follow-up: the two dispatcher encoding tests now patch detect_file_encoding to fail, so a successful decode proves the explicit codec was forwarded instead of relying on BOM-less UTF-16 being unrecoverable by the shared detector.
• Reformatted the _decode_bytes docstring so ruff format --check passes (this was the 3.13 lint failure on the previous head).

Local: pytest test_json.py test_ndjson.py green (169 passed; the lone detect_filetype/libmagic miss is a local environment limitation, unchanged from before), the dispatcher encoding paths were validated standalone with detection patched to raise, and ruff check / ruff format --check are clean on the touched files. Fresh CI runs on this new head.

[cragwolfe]

Re-reviewed exact head c4cebcf3a5a7507f28ea3f9f18ec32ee9a13f448 against current main (3253c2b0f6b0a5f0808910c085675672aff718f3), with a complete production-context GPT Pro review. No actionable introduced source defects remain.

The prior P2 is fixed: JSON forwards encoding into registered chunkers through both rehydration and arbitrary/empty-JSON finalization. The two dispatcher tests now make fallback detection raise, so their successful explicit-codec reads prove forwarding. The integration preserves authoritative explicit decoding, the shared BOM/NUL-aware default detector, read-once acquisition from caller streams, fourth-position metadata_last_modified, and main's JSON identity/metadata processing. Findings and test coverage were checked against source; no local tests were run.

All 51 exact-head checks now pass, including all six required checks and the CI, Security, CodeQL, and Partition Benchmark workflows. This approval covers the reviewed source. The PR remains behind main, and branch protection requires it to be current before merge. After updating the branch, obtain checks on the resulting head before merging.

(authored by codex)

[linhongyu510]

Integration note (no review or merge action taken): merged the latest main (07d978b8) into this branch with an ordinary merge commit a445f41d (fast-forward; no rebase, no force, no --no-verify). Only CHANGELOG.md and __version__.py conflicted; the reviewed encoding source and the regression blobs are unchanged.

Local checks:

• test_json.py + test_ndjson.py: 167 passed, 3 failed.
• Related: common/test_json_partitioning.py + file_utils/test_encoding.py + file_utils/test_encoding_detection_policy.py: 179 passed, 0 failed.

The same 3 failures reproduce on a clean main checkout (absent libmagic for magic MIME detection; Python 3.14 json recursion-depth behavior), so they are pre-existing/environmental rather than introduced here. Ruff (pinned 0.15.2) check and format --check are clean on the changed files.

Fork CI required jobs still need workflow approval on this head. I have not approved, requested changes, resolved a thread, or merged.