Enforce lexical literal constraints from grammar.md

[StreamDemon]

Summary

Four gaps between the lexer and the documented lexical constraints (§16.1, mirrored in docs/reference/grammar.md), from the same review as PRs #70/#72 — all cases where the lexer silently accepted what the grammar calls a compile error:

• Empty character literals: '' lexed as a CharLit. CHAR_LIT requires exactly one Unicode scalar value — now "empty character literal".
• Bare base prefixes: 0x / 0o / 0b with no digits lexed as IntLit. hex_lit/oct_lit/bin_lit each require at least one digit — now e.g. "hexadecimal literal needs at least one digit". (A prefix followed only by _ was already caught by the separator rule and still is.)
• Suffix/base mismatches: integer suffixes on float literals (3.14u32, 1e10u8) silently stayed FloatLit — float_suffix is only f32/f64, so these now error. The mirror case, a float suffix on a based literal (0b1f32, 0o7f64), also errors — FLOAT_LIT only combines a float suffix with dec_lit. (0xFf32 remains a hex int: f, 3, 2 are hex digits, so there is no suffix to validate.)
• Base-aware separator validation: neighbor checks used is_ascii_hexdigit regardless of base, so 1_e5, 1e_5, and 1.5_e3 passed in decimal literals (e is a hex-digit character but not a decimal digit). Neighbors are now checked with the literal's own base; 0xF_F / 0xd_e stay legal.

Deferred, now documented: grammar.md also lists literal overflow as a parse-time error; that needs typed literals, so it stays deferred to semantic analysis — the crates/AGENTS.md subset contract now says so explicitly.

Stacked-PR note: targets the integration branch fix/parser-correctness-base (PR #69), not main — sibling of #70/#72. Only overlap with the siblings is one crates/AGENTS.md list insertion (keep-both at merge); the lexer crate itself is untouched by the other sub-PRs.

Related Issue

None (review-driven; part of the parser correctness wave tracked in PR #69).

Spec Sections Affected

None changed — this PR implements §16.1's lexical constraints as documented in docs/reference/grammar.md.

Checklist

• Code follows the Sploosh design principles (one way to do it, explicit over implicit, etc.)
• Documentation updated in relevant docs/ pages (crates/AGENTS.md subset contract)
• Tests added or updated
• All build targets still compile (if applicable)
• Spec-only PR (skip Build Targets section if checked)

Build Targets Tested

• N/A (docs/spec only) — no codegen targets exist yet; "build" = the Rust workspace.

Test Plan

• 5 new lexer unit tests, one per rule plus the base-awareness matrix: rejects_empty_char_literal (with ' ' and '\'' acceptance), rejects_bare_base_prefixes (all three bases, message names the base), rejects_int_suffix_on_float_literals, rejects_float_suffix_on_based_literals (incl. the 0xFf32 hex-digit-greediness case), separator_validation_is_base_aware (1_e5/1e_5/1.5_e3 rejected; 1_000.5e10, 0xF_F, 0xd_e accepted).
• Existing lexer/parser/corpus tests unchanged and green (1_000u64 in the corpus still lexes).
• cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace green locally and via scripts/docker-check.ps1 -Build (Ubuntu parity).

---

Summary by cubic

Enforces §16.1 literal rules from docs/reference/grammar.md in the lexer so invalid numeric and character literals now error with clear messages. Clarifies in crates/AGENTS.md that overflow checks are deferred to semantic analysis.

• Bug Fixes

  • Reject empty character literals ('') with "empty character literal".
  • Require a digit after 0x/0o/0b; errors name the base.
  • Enforce suffix rules: integer suffix on float literal errors; float suffixes are only f32/f64 and only allowed on decimal; based literals with float suffix error (e.g., 0b1f32), while 0xFf32 remains a hex int.
  • Make numeric separator checks base-aware; reject 1_e5/1e_5/1.5_e3; allow 0xF_F/0xd_e.

• Migration

  • Update code/tests using '', bare base prefixes, int suffixes on floats, or separators touching non-digits in decimal literals.
  • Overflow errors are not emitted yet; they are deferred to semantic analysis (documented in crates/AGENTS.md).

^{Written for commit b48f52a. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 2 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

Architecture diagram

sequenceDiagram
    participant SOURCE as Source Text
    participant LEX as Lexer
    participant TOKENS as Token Stream
    participant GRAMMAR as grammar.md Spec
    
    Note over SOURCE,GRAMMAR: Lexer literal validation flow (§16.1)
    
    LEX->>SOURCE: read next token
    alt Character literal ('')
        SOURCE-->>LEX: '' at position
        LEX->>LEX: char_lit(): scan for scalar
        LEX->>LEX: no scalar found after opening quote
        Note over LEX: NEW: empty character literal
        LEX->>LEX: error("empty character literal")
        LEX-->>TOKENS: no token (error)
    else Numeric literal with base prefix
        SOURCE-->>LEX: 0x / 0o / 0b
        LEX->>LEX: base = ident base(16/8/2)
        LEX->>LEX: digits(base): no digits consumed
        Note over LEX: NEW: bare base prefix
        LEX->>LEX: error("{base} literal needs at least one digit")
        LEX-->>TOKENS: no token (error)
    else Float literal with int suffix
        SOURCE-->>LEX: 3.14u32
        LEX->>LEX: numeric_lit(): detect float kind
        LEX->>LEX: suffix = u32
        Note over LEX: NEW: int suffix on float literal
        alt kind == FloatLit AND suffix starts with 'f'
            LEX->>LEX: validate suffix (f32/f64)
        else kind == FloatLit AND suffix doesn't start with 'f'
            LEX->>LEX: error("float literals only accept `f32`/`f64` suffixes")
        end
        LEX-->>TOKENS: no token (error)
    else Based literal with float suffix
        SOURCE-->>LEX: 0b1f32
        LEX->>LEX: numeric_lit(): base=2
        LEX->>LEX: suffix = f32
        Note over LEX: NEW: float suffix on based literal
        alt suffix starts with 'f' AND base != 10
            LEX->>LEX: error("float suffixes require a decimal literal")
        end
        LEX-->>TOKENS: no token (error)
    else Numeric separator validation
        SOURCE-->>LEX: 1_e5
        LEX->>LEX: numeric_lit(): base=10
        LEX->>LEX: digits(10) -> 1, then '_'
        Note over LEX: NEW: base-aware separator check
        LEX->>LEX: validate_numeric_body(start, base)
        LEX->>LEX: check neighbor: valid_digit('e', base=10)? -> false
        alt neighbor invalid digit for base
            LEX->>LEX: error("numeric separators must appear between digits")
        else neighbors valid (e.g., 0xF_F)
            LEX->>LEX: continue
        end
    else Valid literal
        SOURCE-->>LEX: 0xFF, 3.14f32, etc.
        LEX->>LEX: scan token normally
        LEX-->>TOKENS: token
    end
    
    Note over TOKENS,GRAMMAR: overflow checking deferred to semantic analysis
    LEX->>GRAMMAR: references §16.1 rules
    GRAMMAR-->>LEX: validates constraints

Loading

<sub>Auto-approved: Enforcing lexical literal constraints from grammar.md: empty chars, bare prefixes, suffix mismatches, base-aware separators. Low-risk spec compliance changes with tests.

Re-trigger cubic</sub>