Skip to content

Encode the EUC-JP sources with Python instead of iconv - #773

Closed
quarrel07 wants to merge 1 commit into
n64decomp:masterfrom
quarrel07:macos-euc-jp
Closed

Encode the EUC-JP sources with Python instead of iconv#773
quarrel07 wants to merge 1 commit into
n64decomp:masterfrom
quarrel07:macos-euc-jp

Conversation

@quarrel07

Copy link
Copy Markdown
Contributor

On macOS, make cannot produce a byte-matching ROM for any version. The cause is one line:

$(BUILD_DIR)/%.jp.c: %.c
	$(V)iconv -t EUC-JP -f UTF-8 $< > $@

macOS iconv converts a backslash that follows Japanese text into the fullwidth reverse solidus (0xA1C0) instead of leaving it as 0x5C. GNU iconv leaves it alone, which is why this has never shown up on Linux.

It bites in src/cpu_vehicles_camera_path.c:219:

char* D_800EB710 = "ゴール直後の強制ソート\n";

That trailing \n encodes as:

bytes
GNU iconv / Python 5c 6e (\ n)
macOS iconv a1 c0 6e (fullwidth reverse solidus, then n)

So the escape stops being an escape. Across the file, 26 of the 199 backslashes are converted (0x5C: 199 -> 173, 0xA1C0: 0 -> 26), the generated file grows by exactly 26 bytes, .main grows with it, and the ROM shifts. first-diff.py says:

First difference at ROM addr 0x1008, in entry_point
Instruction difference at ROM addr 0xec328, in build/us/src/cpu_vehicles_camera_path.jp.o
Tons of differences, must be a shifted ROM.

Python's euc_jp codec produces the bytes the cartridge actually has. PYTHON is already a required build dependency. For the other two EUC-JP sources (src/menu_items.c, src/ending/credits.c) macOS iconv and Python output are byte-identical, so this only changes behaviour where iconv was wrong.

Verified on macOS 26.6, Apple Silicon. Before: d2eaf74b99f3f2bc78fa5acc610fcb5e0ce39610. After: 579c48e211ae952530ffc8738709f078d5dd215e, and make reports mk64.us: OK.

Should be a no-op on Linux, since Python emits the same bytes GNU iconv already emits. The matching US ROM is the proof.

macOS iconv converts a backslash that follows Japanese text into the
fullwidth reverse solidus (0xA1C0) rather than leaving it as 0x5C. In
src/cpu_vehicles_camera_path.c that turns a "\n" escape inside a string
literal into a literal two-byte character, adding 26 bytes, which grows
.main and shifts the whole ROM. No version can byte-match on macOS as a
result.

Python's euc_jp codec produces the bytes the cartridge actually has, and
is identical to GNU iconv's output for the other two EUC-JP sources.
PYTHON is already a required build dependency.
@quarrel07

Copy link
Copy Markdown
Contributor Author

Folded into #772.

@quarrel07 quarrel07 closed this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant