[textinput] Support UTF-8 at the prompt - #23085
Conversation
|
Cool! And would fix https://its.cern.ch/jira/browse/ROOT-9562 Maybe also https://root-forum.cern.ch/t/c-20-std-format-fails/ ? |
| // UTF-32 (used inside textinput, where one buffer element must be exactly one | ||
| // character), plus the number of terminal columns a character occupies. | ||
| // | ||
| //===----------------------------------------------------------------------===// |
There was a problem hiding this comment.
was this file imported from llvm?
There was a problem hiding this comment.
It's the same header as in the other textinput sources. It seems to me that the original author of textinput, @axel, wanted to maybe upstream it to LLVM at some point, and therefore mimicked LLVMs license. Do you know the history?
There was a problem hiding this comment.
There was a problem hiding this comment.
If CERN people were the only contributors then this file should be (c) CERN and you should be able to freely change the license for future versions of this file.
| // This file is distributed under the University of Illinois Open Source | ||
| // License. See LICENSE.TXT for details. | ||
| // | ||
| //===----------------------------------------------------------------------===// |
There was a problem hiding this comment.
was this file imported from llvm?
|
this is remarkable. |
96498a7 to
06fbd6d
Compare
Test Results 23 files 23 suites 3d 18h 22m 38s ⏱️ For more details on these failures, see this check. Results for commit 7c5883e. ♻️ This comment has been updated with latest results. |
|
🎉 Great! Now we are one step closer to being able to count past 255. 🚀 |
06fbd6d to
e110a60
Compare
textinput assumed one line-buffer element is one byte and one terminal column. Both assumptions break on UTF-8, so anything outside ASCII corrupted the buffer on input and misplaced the cursor on output. Text now stores std::u32string, so every index in the library counts characters; the editor's existing index arithmetic (cursor motions, cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8, cached with a character-to-byte offset table, so Getline's C interface, the interpreter and the history file are unaffected. Since a character occupies 0, 1 or 2 terminal columns, the display now lays out lines by accumulated character width (CharWidth() in the new UTF8.h) instead of index arithmetic, and moves a double-width character that would straddle the margin to the next line. The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8 sequences (previously bytes >= 0x80 went negative, colliding with the EOF sentinel and hitting undefined isprint()); StreamReaderWin reads through ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key and AltGr events it used to drop, and TerminalDisplayWin writes through WriteConsoleW. Also fixed: byte-vs-character index mixups in HandleControl(), Getline_color.cxx and tab completion (whose byte-space cursor could end up past the end of the line and abort), a pre-existing wrong-variable bug in the colorizer's trailing-space trim, and stale screen contents when deleting a combining mark. Verified on Linux with unit tests for the encoding/width layer and by driving root.exe through a pty: painting, cursor columns, wrapping, editing, history, reverse search and tab completion over accented, combining, CJK and emoji input. The Windows sources are only type-checked, not built. 🤖 Done with the help of AI.
e110a60 to
7c5883e
Compare
5 participants
Experiment requested by @dpiparo
textinput assumed one line-buffer element is one byte and one terminal column. Both assumptions break on UTF-8, so anything outside ASCII corrupted the buffer on input and misplaced the cursor on output.
Text now stores std::u32string, so every index in the library counts characters; the editor's existing index arithmetic (cursor motions, cut/paste, ...) becomes correct unchanged. GetText() still returns UTF-8, cached with a character-to-byte offset table, so Getline's C interface, the interpreter and the history file are unaffected. Since a character occupies 0, 1 or 2 terminal columns, the display now lays out lines by accumulated character width (CharWidth() in the new UTF8.h) instead of index arithmetic, and moves a double-width character that would straddle the margin to the next line.
The readers now decode multi-byte input: StreamReaderUnix assembles UTF-8 sequences (previously bytes >= 0x80 went negative, colliding with the EOF sentinel and hitting undefined isprint()); StreamReaderWin reads through ReadConsoleInputW, combining surrogate pairs and keeping the IME, dead-key and AltGr events it used to drop, and TerminalDisplayWin writes through WriteConsoleW. Also fixed: byte-vs-character index mixups in HandleControl(), Getline_color.cxx and tab completion (whose byte-space cursor could end up past the end of the line and abort), a pre-existing wrong-variable bug in the colorizer's trailing-space trim, and stale screen contents when deleting a combining mark.
Verified on Linux with unit tests for the encoding/width layer and by driving root.exe through a pty: painting, cursor columns, wrapping, editing, history, reverse search and tab completion over accented, combining, CJK and emoji input. The Windows sources are only type-checked, not built.
🤖 Done with the help of AI.
Experiment requested by @dpiparo
Fixes ROOT-9562