Found while fixing #9421 (three separate bugs; this is the third). Stress case D04_p_binary_stdin.
The defect
Feeding bytes 0..255 through a stream with .setEncoding("utf8"):
|
code units out |
U+FFFD count |
high bytes |
| node |
256 |
128 |
replaced per WHATWG |
| perry |
158 |
0 |
raw U+0080..U+00FF pass through |
Two failures at once: invalid UTF-8 is not replaced with U+FFFD, and 98 units simply vanish. Buffer.toString("utf8") is correct in perry — the defect is specifically the stream decoder (which must also handle multi-byte sequences split across chunk boundaries; suspect that machinery).
The user-visible damage
cc writes what it read into the session transcript: perry's JSONL lines contain raw 0x80..0xFF bytes — neither valid UTF-8 nor parseable JSON — so --resume cannot read the session back. Record count matches node (7 vs 7); the file is corrupt, not short. The 16,717 → 12,721 byte difference is the missing \xef\xbf\xbd escaping, not lost records.
Fix shape
Route the stream decoder through the same correct machinery Buffer.toString("utf8") uses, with carry-over state for sequences split across chunks (node's string_decoder semantics: emit U+FFFD per WHATWG for invalid sequences, hold incomplete trailing sequences until the next chunk or end).
Verification bar
Gap fixture: all 256 byte values in one chunk; a 4-byte emoji split at every possible chunk boundary (1/3, 2/2, 3/1); a lone continuation byte; an overlong encoding; truncated sequence at end. Assert code-unit counts and exact replacement positions against node. Then the cc check: D04 transcript lines all JSON.parse-able and byte-identical to node's.
Measurements by the #9421 investigation; repro artifacts /root/claude-9421/ on the build host.
Found while fixing #9421 (three separate bugs; this is the third). Stress case
D04_p_binary_stdin.The defect
Feeding bytes 0..255 through a stream with
.setEncoding("utf8"):Two failures at once: invalid UTF-8 is not replaced with U+FFFD, and 98 units simply vanish.
Buffer.toString("utf8")is correct in perry — the defect is specifically the stream decoder (which must also handle multi-byte sequences split across chunk boundaries; suspect that machinery).The user-visible damage
cc writes what it read into the session transcript: perry's JSONL lines contain raw
0x80..0xFFbytes — neither valid UTF-8 nor parseable JSON — so--resumecannot read the session back. Record count matches node (7 vs 7); the file is corrupt, not short. The 16,717 → 12,721 byte difference is the missing\xef\xbf\xbdescaping, not lost records.Fix shape
Route the stream decoder through the same correct machinery
Buffer.toString("utf8")uses, with carry-over state for sequences split across chunks (node'sstring_decodersemantics: emit U+FFFD per WHATWG for invalid sequences, hold incomplete trailing sequences until the next chunk orend).Verification bar
Gap fixture: all 256 byte values in one chunk; a 4-byte emoji split at every possible chunk boundary (1/3, 2/2, 3/1); a lone continuation byte; an overlong encoding; truncated sequence at
end. Assert code-unit counts and exact replacement positions against node. Then the cc check:D04transcript lines allJSON.parse-able and byte-identical to node's.Measurements by the #9421 investigation; repro artifacts
/root/claude-9421/on the build host.