fix(protocol): advertise ISO-8859-1, not UTF-8, in CHARSET negotiation - #278
Open
ahumbert wants to merge 1 commit into
Open
fix(protocol): advertise ISO-8859-1, not UTF-8, in CHARSET negotiation#278ahumbert wants to merge 1 commit into
ahumbert wants to merge 1 commit into
Conversation
On a client's WILL CHARSET, the server replied with a REQUEST for UTF-8 -- but it has no UTF-8 output path at all. ProtocolOutput(), the only function that would encode one (via UnicodeGet()), has no callers anywhere in the server; text goes straight from process_output() to write_to_descriptor() as raw bytes. The server's only non-ASCII output is the Quenya month names in consts.cpp (month_name[0..11], e.g. "N\xE9nim\xEB"), reached via day_to_str() from the `time` command and the pkill log. Those are literal latin-1 bytes. A scan of the live data confirms nothing else is affected: zero high-bit bytes across all 1479 world files, lib/text, lib/misc, lib/boards, 33919 player files and 5931 plrobjs files. So conforming clients were told UTF-8 and handed latin-1. Mudlet decoded a lone 0xEB as an invalid UTF-8 sequence and rendered the month names as replacement characters. Declaring ISO-8859-1 makes the advertisement match what we actually emit, and since everything else is plain ASCII, no other output changes. Also stop setting eMSDP_UTF_8 on an ACCEPTED reply. That flag means "UTF-8 is in play"; accepting latin-1 is the opposite, so asserting it there would now be wrong in the other direction. Genuine client-side UTF-8 capability is still recorded from the MTTS bitfield in the TTYPE handler, which describes the client rather than this negotiation. Not addressed here: actually emitting UTF-8 would be a much larger change, because process_output()'s PRF_LATIN1 fold calls unaccent() per byte. Given UTF-8 month names it would map 0xC3 to 'A' and 0xAB to 255, turning "N\xE9nim\xEB" into garbage for every player with latin-1 disabled. That path has to become encoding-aware first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6cxogr7Tuk1qU84M3knW4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a client's WILL CHARSET, the server replied with a REQUEST for UTF-8 -- but it has no UTF-8 output path at all. ProtocolOutput(), the only function that would encode one (via UnicodeGet()), has no callers anywhere in the server; text goes straight from process_output() to write_to_descriptor() as raw bytes.
The server's only non-ASCII output is the Quenya month names in consts.cpp (month_name[0..11], e.g. "N\xE9nim\xEB"), reached via day_to_str() from the
timecommand and the pkill log. Those are literal latin-1 bytes. A scan of the live data confirms nothing else is affected: zero high-bit bytes across all 1479 world files, lib/text, lib/misc, lib/boards, 33919 player files and 5931 plrobjs files.So conforming clients were told UTF-8 and handed latin-1. Mudlet decoded a lone 0xEB as an invalid UTF-8 sequence and rendered the month names as replacement characters. Declaring ISO-8859-1 makes the advertisement match what we actually emit, and since everything else is plain ASCII, no other output changes.
Also stop setting eMSDP_UTF_8 on an ACCEPTED reply. That flag means "UTF-8 is in play"; accepting latin-1 is the opposite, so asserting it there would now be wrong in the other direction. Genuine client-side UTF-8 capability is still recorded from the MTTS bitfield in the TTYPE handler, which describes the client rather than this negotiation.
Not addressed here: actually emitting UTF-8 would be a much larger change, because process_output()'s PRF_LATIN1 fold calls unaccent() per byte. Given UTF-8 month names it would map 0xC3 to 'A' and 0xAB to 255, turning "N\xE9nim\xEB" into garbage for every player with latin-1 disabled. That path has to become encoding-aware first.
Claude-Session: https://claude.ai/code/session_01K6cxogr7Tuk1qU84M3knW4