Skip to content

fix(protocol): advertise ISO-8859-1, not UTF-8, in CHARSET negotiation - #278

Open
ahumbert wants to merge 1 commit into
returnoftheshadow:release-frodofrom
ahumbert:charset-iso-8859-1
Open

fix(protocol): advertise ISO-8859-1, not UTF-8, in CHARSET negotiation#278
ahumbert wants to merge 1 commit into
returnoftheshadow:release-frodofrom
ahumbert:charset-iso-8859-1

Conversation

@ahumbert

@ahumbert ahumbert commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

On a client's WILL CHARSET, the server replied with a REQUEST for UTF-8 -- but it has no UTF-8 output path at all. ProtocolOutput(), the only function that would encode one (via UnicodeGet()), has no callers anywhere in the server; text goes straight from process_output() to write_to_descriptor() as raw bytes.

The server's only non-ASCII output is the Quenya month names in consts.cpp (month_name[0..11], e.g. "N\xE9nim\xEB"), reached via day_to_str() from the time command and the pkill log. Those are literal latin-1 bytes. A scan of the live data confirms nothing else is affected: zero high-bit bytes across all 1479 world files, lib/text, lib/misc, lib/boards, 33919 player files and 5931 plrobjs files.

So conforming clients were told UTF-8 and handed latin-1. Mudlet decoded a lone 0xEB as an invalid UTF-8 sequence and rendered the month names as replacement characters. Declaring ISO-8859-1 makes the advertisement match what we actually emit, and since everything else is plain ASCII, no other output changes.

Also stop setting eMSDP_UTF_8 on an ACCEPTED reply. That flag means "UTF-8 is in play"; accepting latin-1 is the opposite, so asserting it there would now be wrong in the other direction. Genuine client-side UTF-8 capability is still recorded from the MTTS bitfield in the TTYPE handler, which describes the client rather than this negotiation.

Not addressed here: actually emitting UTF-8 would be a much larger change, because process_output()'s PRF_LATIN1 fold calls unaccent() per byte. Given UTF-8 month names it would map 0xC3 to 'A' and 0xAB to 255, turning "N\xE9nim\xEB" into garbage for every player with latin-1 disabled. That path has to become encoding-aware first.

Claude-Session: https://claude.ai/code/session_01K6cxogr7Tuk1qU84M3knW4

On a client's WILL CHARSET, the server replied with a REQUEST for UTF-8 --
but it has no UTF-8 output path at all. ProtocolOutput(), the only function
that would encode one (via UnicodeGet()), has no callers anywhere in the
server; text goes straight from process_output() to write_to_descriptor()
as raw bytes.

The server's only non-ASCII output is the Quenya month names in consts.cpp
(month_name[0..11], e.g. "N\xE9nim\xEB"), reached via day_to_str() from the
`time` command and the pkill log. Those are literal latin-1 bytes. A scan of
the live data confirms nothing else is affected: zero high-bit bytes across
all 1479 world files, lib/text, lib/misc, lib/boards, 33919 player files and
5931 plrobjs files.

So conforming clients were told UTF-8 and handed latin-1. Mudlet decoded a
lone 0xEB as an invalid UTF-8 sequence and rendered the month names as
replacement characters. Declaring ISO-8859-1 makes the advertisement match
what we actually emit, and since everything else is plain ASCII, no other
output changes.

Also stop setting eMSDP_UTF_8 on an ACCEPTED reply. That flag means "UTF-8
is in play"; accepting latin-1 is the opposite, so asserting it there would
now be wrong in the other direction. Genuine client-side UTF-8 capability is
still recorded from the MTTS bitfield in the TTYPE handler, which describes
the client rather than this negotiation.

Not addressed here: actually emitting UTF-8 would be a much larger change,
because process_output()'s PRF_LATIN1 fold calls unaccent() per byte. Given
UTF-8 month names it would map 0xC3 to 'A' and 0xAB to 255, turning
"N\xE9nim\xEB" into garbage for every player with latin-1 disabled. That
path has to become encoding-aware first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6cxogr7Tuk1qU84M3knW4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant