Allow non-ASCII characters in identifiers#3028
Draft
ksss wants to merge 1 commit into
Draft
Conversation
Ruby accepts identifiers with non-ASCII code points — a class name like `Ú` (U+00DA) or `Ω` (U+03A9) is a valid constant, and a method name like `únicos` or `日本語` is a valid local identifier. The RBS lexer, however, recognised identifiers only when the surrounding characters were ASCII (`[a-zA-Z0-9_]`), so writing the same names in a signature file failed to parse. rubocop-on-rbs issue ruby#151 tracks this on the linter side. Extend the lexer's identifier grammar to accept any valid multibyte character as either the leading or a continuation code point. The lexer already maps every non-ASCII code point to a fixed sentinel (U+30EB) via `rbs_next_char`, so a single `mb_char = [ル]` class covers all multibyte characters. When the leading character is multibyte, the token is dispatched to `rbs_next_mb_ident_token`, which reads the real bytes of the identifier's first code point and asks the encoding's `isupper_char` whether it belongs to Unicode's `Uppercase` property — the same test Ruby's `rb_sym_constant_char_p` uses to classify a constant. That decides `tUIDENT` vs. `tLIDENT` (or `tULIDENT` vs. `tULLIDENT` for the `_`-prefixed variants). Depends on the invalid-byte-to-U+FFFD sentinel introduced in ruby#2983: the `mb_char` class matches U+30EB (valid multibyte) but not U+FFFD (invalid byte), so an invalid UTF-8 byte inside an identifier still surfaces as an ErrorToken rather than being silently swallowed. Align `TypeName#kind` on both the Ruby and Rust sides with the same Unicode semantics. Ruby now falls through to `\p{Uppercase}` for non-ASCII code points after the ASCII fast paths, and Rust decodes the first code point with `char::is_uppercase` only when the leading byte is non-ASCII. ASCII names — the overwhelming majority — keep their current single-comparison cost on both sides. test/rbs/type_parsing_test.rb's `test_parse__byte_range_incorrect` documented an incidental side effect where starting `byte_range` in the middle of a multibyte character produced an ErrorToken. That was only true because non-ASCII was rejected wholesale; the lexer already rounds `start_pos` up to the next character boundary, so the parse now succeeds on the next valid identifier. Rewrite the test to describe the actual behaviour and to demonstrate how `require_eof: true` still catches the leftover trailing token. src/lexer.c is regenerated with re2c 4.3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ruby accepts identifiers with non-ASCII code points — a class name like
Ú(U+00DA) orΩ(U+03A9) is a valid constant, and a method name likeúnicosor日本語is a valid local identifier. The RBS lexer, however, only recognised identifiers when the surrounding characters were ASCII ([a-zA-Z0-9_]), so writing the same names in a signature file failed to parse. See rubocop-on-rbs#151 for the linter-side motivation.This PR:
TypeName#kindon the Ruby and Rust sides with the same UnicodeUppercasesemantics the lexer now uses.Lexer
rbs_next_charalready maps every non-ASCII code point to a fixed sentinelU+30EB, so a singlemb_char = [ル]class covers all multibyte characters.wordand theidentifieralias are widened to includemb_char, so existing rules like[a-z] word*and"@" [a-zA-Z_] word*automatically accept trailing multibyte code points without changing their token kind.Two new rules handle a multibyte leading code point:
rbs_next_mb_ident_tokenreads the real bytes of the identifier's first code point and asks the encoding'sisupper_charwhether it belongs to Unicode'sUppercaseproperty — the same test Ruby'srb_sym_constant_char_puses to decideconstantvs.local. That pickstUIDENTvs.tLIDENT(ortULIDENTvs.tULLIDENTfor the_-prefixed variants).Depends on #2983
The double-sentinel design introduced in #2983 (valid multibyte →
U+30EB, invalid byte →U+FFFD) is what makes this change safe:mb_charmatches onlyU+30EB, so an invalid UTF-8 byte inside an identifier still surfaces as anErrorTokenrather than being silently absorbed into the token.Behaviour changes
Given the semantics above, the following now parse:
class ServicioÚltimaVez,class Ωmega(Unicode uppercase →tUIDENT)def enviar_últimas_interacciones,def 日本語(any letter →tLIDENT)@日本語,@@クラス変数,$グローバルThe following are still rejected, matching Ruby:
class αlpha— Greekαis Unicode lowercase, so the token istLIDENTand the parser rejects it as a class name.class 日本語— kanji is Other_Letter (a local identifier in Ruby), same story.ErrorToken.TypeName#kind
TypeName#kindpreviously classifiedÚand日本語as:classonly because it fell through the default branch.αlphaalso landed on:classfor the same accidental reason, which was wrong.Both sides now use the same Unicode
Uppercaseproperty the lexer uses, keeping ASCII on a fast path:lib/rbs/type_name.rb) checks[A-Z]/[a-z]first, then\p{Uppercase}for non-ASCII code points.rust/ruby-rbs/src/type_name.rs) checks the leading byte withis_ascii_uppercasewhen it is ASCII, else decodes one code point withchars().next()and callschar::is_uppercase.ASCII names — the overwhelming majority — pay the same single comparison as before. Non-ASCII names pay one code point decode plus a Unicode table lookup.
src/lexer.cis regenerated with re2c 4.3.