Skip to content

stream.setEncoding('utf8') mis-decodes: bytes 0x80-0xFF pass through raw (node: U+FFFD), and data is LOST — cc transcripts become unparseable JSONL #9490

Description

@proggeramlug

Found while fixing #9421 (three separate bugs; this is the third). Stress case D04_p_binary_stdin.

The defect

Feeding bytes 0..255 through a stream with .setEncoding("utf8"):

code units out U+FFFD count high bytes
node 256 128 replaced per WHATWG
perry 158 0 raw U+0080..U+00FF pass through

Two failures at once: invalid UTF-8 is not replaced with U+FFFD, and 98 units simply vanish. Buffer.toString("utf8") is correct in perry — the defect is specifically the stream decoder (which must also handle multi-byte sequences split across chunk boundaries; suspect that machinery).

The user-visible damage

cc writes what it read into the session transcript: perry's JSONL lines contain raw 0x80..0xFF bytes — neither valid UTF-8 nor parseable JSON — so --resume cannot read the session back. Record count matches node (7 vs 7); the file is corrupt, not short. The 16,717 → 12,721 byte difference is the missing \xef\xbf\xbd escaping, not lost records.

Fix shape

Route the stream decoder through the same correct machinery Buffer.toString("utf8") uses, with carry-over state for sequences split across chunks (node's string_decoder semantics: emit U+FFFD per WHATWG for invalid sequences, hold incomplete trailing sequences until the next chunk or end).

Verification bar

Gap fixture: all 256 byte values in one chunk; a 4-byte emoji split at every possible chunk boundary (1/3, 2/2, 3/1); a lone continuation byte; an overlong encoding; truncated sequence at end. Assert code-unit counts and exact replacement positions against node. Then the cc check: D04 transcript lines all JSON.parse-able and byte-identical to node's.

Measurements by the #9421 investigation; repro artifacts /root/claude-9421/ on the build host.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions