Encode the EUC-JP sources with Python instead of iconv - #773
Closed
quarrel07 wants to merge 1 commit into
Closed
Conversation
macOS iconv converts a backslash that follows Japanese text into the fullwidth reverse solidus (0xA1C0) rather than leaving it as 0x5C. In src/cpu_vehicles_camera_path.c that turns a "\n" escape inside a string literal into a literal two-byte character, adding 26 bytes, which grows .main and shifts the whole ROM. No version can byte-match on macOS as a result. Python's euc_jp codec produces the bytes the cartridge actually has, and is identical to GNU iconv's output for the other two EUC-JP sources. PYTHON is already a required build dependency.
Contributor
Author
|
Folded into #772. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On macOS,
makecannot produce a byte-matching ROM for any version. The cause is one line:macOS
iconvconverts a backslash that follows Japanese text into the fullwidth reverse solidus (0xA1C0) instead of leaving it as0x5C. GNU iconv leaves it alone, which is why this has never shown up on Linux.It bites in
src/cpu_vehicles_camera_path.c:219:That trailing
\nencodes as:5c 6e(\n)a1 c0 6e(fullwidth reverse solidus, thenn)So the escape stops being an escape. Across the file, 26 of the 199 backslashes are converted (
0x5C: 199 -> 173,0xA1C0: 0 -> 26), the generated file grows by exactly 26 bytes,.maingrows with it, and the ROM shifts.first-diff.pysays:Python's
euc_jpcodec produces the bytes the cartridge actually has.PYTHONis already a required build dependency. For the other two EUC-JP sources (src/menu_items.c,src/ending/credits.c) macOS iconv and Python output are byte-identical, so this only changes behaviour where iconv was wrong.Verified on macOS 26.6, Apple Silicon. Before:
d2eaf74b99f3f2bc78fa5acc610fcb5e0ce39610. After:579c48e211ae952530ffc8738709f078d5dd215e, and make reportsmk64.us: OK.Should be a no-op on Linux, since Python emits the same bytes GNU iconv already emits. The matching US ROM is the proof.