Skip to content

Render text as UTF-8 so non-Latin scripts can be displayed - #418

Open
jyh9521 wants to merge 2 commits into
GooberRF:masterfrom
jyh9521:af/feat-utf8-text
Open

Render text as UTF-8 so non-Latin scripts can be displayed#418
jyh9521 wants to merge 2 commits into
GooberRF:masterfrom
jyh9521:af/feat-utf8-text

Conversation

@jyh9521

@jyh9521 jyh9521 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

GrNewFont indexed glyphs with char_map_[256], one byte per glyph, so any text
outside Windows-1252 came out as per-byte mojibake and a translation into a
non-Latin script wasn't possible at all. This keys the glyph map by Unicode code
point and decodes UTF-8 in draw() and get_size().

Legacy text is unaffected. Bytes 0x80-0xFF are invalid UTF-8 lead bytes, so
the decoder returns them unchanged and they're translated from Windows-1252 when
the glyph is rasterized — the stock English, German and French tables render
exactly as before.

Glyphs are rasterized on demand

Pre-baking a character set large enough for Chinese costs 124 MB of atlas at
4K, plus the same again in video memory, in a 32 bit process. Measured across the
seven font instances Alpine creates, against my own translation's 1434 code
points:

atlas
1280x720 37 MB
1920x1080 52 MB
2560x1440 64 MB
3840x2160 124 MB

With no extra characters it's 7.6 MB, so essentially all of it is glyphs a given
screen never shows. Each font now fills page sized atlases with a shelf allocator
and creates another when they're full, so the cost tracks what's actually drawn.

A page keeps its own copy of its pixels: locking a bitmap WRITE_ONLY hands back
staging memory that isn't seeded from the current texture and unlocking copies the
whole surface back, so writing one glyph in place would erase every glyph already
in the page. Pages upload whole, at most once per string drawn.

.vf fonts resolve to their TrueType equivalents

Alpine already swaps the four rf::ui globals for TrueType, but widgets that hold
their own font handle — the weapon priority list and the control binding list
among them — still ask for a .vf, and a bitmap font can't render the text no
matter what the rest of the UI does. The three general purpose text fonts are
redirected; bigfont.vf, smallfont.vf and biggerfont.vf are left alone, as
they hold 32-69 glyphs with a non-standard first_ascii and are special purpose
glyph sets for HUD counters rather than general text.

The replacement is the same size as the font it stands in for, so only the glyph
source changes and everything laid out around those fonts keeps its proportions.
Scaling here instead would blow up every HUD element that is sized for the small
bitmap font when Big HUD is off, and it would tie the redirect to a scale that
isn't known yet during start-up — font ids are handed out once and cached by the
caller for good, so the same .vf would resolve to two different fonts and the
engine would then measure a string with one and draw it with the other. Both of
those were real, and took a while to track down.

Line breaking and truncation

gr_split_str breaks on whole code points and knows the usual CJK rules: a line
may break between two ideographs, but never before closing punctuation nor after
opening punctuation. Latin space breaking is unchanged. gr_fit_string and the
HUD string truncation likewise trim whole code points, so a multi-byte character
is never cut in half.


Tested at 3840x2160, Big HUD on and off, stock English and with a Simplified
Chinese translation loaded: menus, HUD, subtitles, message log, line breaking and
truncation.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Alpine Faction’s TrueType font rendering pipeline to properly handle UTF-8 text, enabling non‑Latin scripts (e.g., CJK) to render correctly in-game. It also changes glyph management to rasterize glyphs on demand (paged atlases) and adds CJK-aware line breaking and code-point-safe truncation, while keeping legacy Windows‑1252 text working.

Changes:

  • Decode UTF‑8 code points in text drawing/measurement and key glyph caching by Unicode code point.
  • Rasterize glyphs lazily into shelf-packed atlas “pages” to avoid large up-front atlas allocations.
  • Add CJK-aware line breaking and ensure truncation/ellipsis operate on whole code points.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
game_patch/hud/hud.cpp Updates HUD ellipsis fitting to avoid splitting multi-byte characters.
game_patch/graphics/gr_font.cpp Implements UTF‑8 decoding, lazy glyph rasterization into paged atlases, .vf→TTF redirects, CJK-aware splitting, and code-point-safe fit logic.
docs/CHANGELOG.md Documents the UTF‑8 rendering + lazy glyph rasterization feature set for the upcoming release.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread game_patch/graphics/gr_font.cpp Outdated
Comment thread game_patch/hud/hud.cpp Outdated
@jyh9521
jyh9521 force-pushed the af/feat-utf8-text branch from ffca388 to e4b1be3 Compare August 13, 2026 00:23
GrNewFont indexed glyphs with char_map_[256], one byte per glyph, so any text
outside Windows-1252 came out as per-byte mojibake and a translation into a
non-Latin script was not possible. Key the glyph map by Unicode code point
instead and decode UTF-8 in draw() and get_size().

Legacy text keeps working unchanged. Bytes 0x80-0xFF are invalid UTF-8 lead
bytes, so the decoder returns them as-is and they are translated from
Windows-1252 when the glyph is rasterized. The stock English, German and French
tables render exactly as before.

Glyphs are rasterized on first use rather than up front. Pre-baking a character
set large enough for Chinese costs 124 MB of atlas at 4K, plus the same again in
video memory, in a 32 bit process; a screen of text uses a couple of hundred
distinct characters. Each font fills page sized atlases with a shelf allocator
and creates another when they are full. A page keeps its own copy of its pixels
because locking a bitmap WRITE_ONLY hands back staging memory that is not seeded
from the current texture, so a partial write would erase the glyphs already
there; pages are uploaded whole, at most once per string drawn.

The three general purpose .vf fonts are redirected to their TrueType
equivalents. Alpine already swaps the four rf::ui globals, but widgets that hold
their own font handle - the weapon priority list and the control binding list
among them - still ask for a .vf, and a bitmap font cannot render the text no
matter what the rest of the UI does. bigfont.vf, smallfont.vf and biggerfont.vf
are left alone: they hold 32-69 glyphs with a non-standard first_ascii and are
special purpose glyph sets for HUD counters, not general text.

The replacement is the same size as the font it stands in for, so only the glyph
source changes and everything laid out around those fonts keeps its proportions.
Scaling here instead would blow up every HUD element that is sized for the small
bitmap font when Big HUD is off, and would tie the redirect to a scale that is
not known yet during start-up - font ids are handed out once and cached by the
caller for good, so the same .vf would resolve to two different fonts and the
engine would measure a string with one and draw it with the other.

gr_split_str is replaced with an implementation that breaks on whole code points
and knows the usual CJK rules: a line may break between two ideographs, but
never before closing punctuation nor after opening punctuation. Latin space
breaking is unchanged. gr_fit_string and the HUD string truncation likewise trim
whole code points, so a multi-byte character is never cut in half.
@jyh9521
jyh9521 force-pushed the af/feat-utf8-text branch from e4b1be3 to 0ecc63a Compare August 20, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants