Skip to content

Escape U+0085, U+2028 and U+2029 in NDJSON records - #86

Merged
sidkmenon merged 2 commits into
mainfrom
sm/malf
Oct 5, 2026
Merged

sidkmenon merged 2 commits into
mainfrom
sm/malf

Conversation

@sidkmenon

@sidkmenon sidkmenon commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

JSON allows U+2028 (LINE SEPARATOR) and U+2029 (PARAGRAPH SEPARATOR) raw inside strings, so diffr wrote them raw when a diffed file held one. Since Node 24, node:readline also ends a line at these characters, so a client that reads diffr with readline gets a record cut in two and fails to parse it. Whiteboard's structural diff stops with Malformed diffr protocol record. (devdotfast/whiteboard#624).

Python's str.splitlines() has the same fault for U+0085 (NEL). Every other character it splits on is a control character that JSON already escapes.

  • Adds protocol::write_record. Its formatter writes the same compact JSON, and escapes the three characters as JSON \uXXXX escapes. The decoded text and its byte columns do not change.
  • run.rs writes every record through it.
  • The protocol docs now say that records end at \n only and that the three characters are escaped.

Testing

  • New unit test: the three characters are escaped, and the record parses back to the same event.
  • The issue's repro on Node 24.18: the output has no raw U+2028, and readline reads 3 records that all parse.
  • A file that holds all nine characters readline or splitlines() end a line at: splitlines() reads 3 records that all parse.
  • cargo test, cargo clippy --all-targets (no new warnings) and cargo fmt --all -- --check passed locally.

AI assistance: written with Claude Code; reviewed by the author.

@sidkmenon sidkmenon changed the title Escape U+2028 and U+2029 in NDJSON records Escape U+0085, U+2028 and U+2029 in NDJSON records Oct 5, 2026
JSON allows both characters raw inside strings, so diffr wrote them raw
when a diffed file held one. Since Node 24, node:readline also ends a
line at them, so a client reading diffr with readline got a record cut
in two and failed to parse it (devdotfast/whiteboard#624). Records are
now written through protocol::write_record, whose formatter escapes
both as \u2028 and \u2029. The decoded text is unchanged.

AI assistance: written with Claude Code; reviewed by the author.

Agent-Session: 82924cb5-dcfa-42e4-abb0-ef40393249f8
Agent-Session: ea775653-fef8-47be-96ea-16815bdff22c
Agent-Session: d72470a3-1f93-4201-8e6d-2730c798e92b
Python's str.splitlines() also ends a line at U+0085 (NEL), which serde_json
writes raw. Every other character it splits on is a control character JSON
already escapes, so records now survive splitlines() whole.

AI assistance: written with Claude Code; reviewed by the author.

Agent-Session: ea775653-fef8-47be-96ea-16815bdff22c
Agent-Session: d72470a3-1f93-4201-8e6d-2730c798e92b
@sidkmenon
sidkmenon merged commit 5120b24 into main Oct 5, 2026
48 of 49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant