DEV-1833: harden Mode-B keyword lexing (CASE/LIKE) for Unicode identifiers

docs/architecture/parsing.md

@@ -121,8 +121,7 @@ Filters historically accepted SQL operator spellings (`=`, `<>`, `NULL`, keyword
 `AND`/`OR`/`NOT`/`IS`/`IN`) alongside Python ones. `parse_filter_expr` normalizes
 those to Python equivalents (string-literal-aware) via
 `_normalize_sql_filter_operators`, then delegates to `parse_expr`. Measures and
-order parse with `parse_expr` directly; only filters get the leniency — matching
-the legacy `parse_filter` contract.
+order parse with `parse_expr` directly; only filters get the leniency.
 
 ### `walk_parsed_refs` — scope-free reference extraction
 

docs/concepts/formulas.md

@@ -526,6 +526,9 @@ Any formula, filter, or field expression can branch with SQL `CASE`:
 - **Searched** (`CASE WHEN c1 THEN v1 [WHEN c2 THEN v2 …] [ELSE d] END`) and
   **simple** (`CASE x WHEN v1 THEN r1 … END`, lowered to `x = v1`) forms are both
   accepted; keywords are case-insensitive and CASE nests anywhere.
+- Identifiers named after, containing, or qualified by SQL keywords (`case`,
+  `customers.end`, `écase`) always parse as ordinary references — `CASE` starts
+  a conditional only when a `WHEN` follows it.
 - A missing `ELSE` yields `NULL`. `iif(cond, then, otherwise)` is an equivalent
   spelling — an allowlisted scalar function taking exactly three arguments.
   Everything renders to a portable SQL `CASE` on every Tier-1 dialect.

openspec/changes/archive/2026-09-04-dev-1833-harden-mode-b-keyword-lexing-case-like-for-unicode/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-04

openspec/changes/archive/2026-09-04-dev-1833-harden-mode-b-keyword-lexing-case-like-for-unicode/design.md

@@ -0,0 +1,86 @@
+# Design
+
+## Context
+
+See proposal.md — Why. The rewriter layer at stake runs on the raw Mode-B string before
+`ast.parse`: `_rewrite_case_when` (all expressions, `slayer/engine/syntax.py:420`) and
+`_rewrite_sql_like` + `_normalize_sql_filter_operators` (filters only). Verified current
+failures: bare `case` / `customers.case` / `écase` raise "Malformed CASE"; `customers.end`
+in a THEN value yields `iif(a, customers., None) ELSE 0 END` (silent corruption);
+`_rewrite_sql_like` rewrites inside string literals and truncates escaped-quote patterns.
+The legacy `formula.parse_filter` has one production caller
+(`schema_drift._filter_refs_dsl`, already `try/except → []`); its `.sql` output is
+production-dead since DEV-1450. DEV-1826 makes both aggregation spellings parse natively
+in the typed parser (verified: `sum(revenue) > 100`, `revenue:countd > 5`, `*:count > 3`
+all yield `AggCall`), so the migration needs no funcstyle pre-rewrite.
+
+## Goals / Non-Goals
+
+Goals: keyword-safe lexing per the spec; exactly one Mode-B filter parser and one LIKE
+rewriter in the codebase. Non-goals: Unicode support in `_COLON_AGG_RE` /
+`_SCAN_TOKEN_RE` (ASCII-start colon-agg identifiers — separate issue if wanted);
+double-quoted LIKE patterns; migrating converters off `parse_formula`.
+
+## Decisions
+
+1. **Tokenizer** (`_CASE_TOKEN_RE` id alternative):
+   `[^\W\d]\w*(?:\s*\.\s*[^\W\d]\w*)*` — Unicode-aware start; dotted paths (whitespace
+   tolerated around dots, matching Python attribute syntax) lex as ONE token, so a
+   qualified keyword can never equal a keyword. Alternative rejected: `tokenize` module
+   (the text is not valid Python at this stage — SQL keywords, colon aggs).
+2. **Keyword guard** (helper applied at every comparison site): a token is a keyword iff
+   it `isascii()` and uppercases to the keyword AND neither adjacent raw-text character
+   is identifier material beyond `\w` — i.e. not `isalnum()`/`_`/`isidentifier()` and not
+   Unicode category `M*` (combining marks). Covers `caſe`, decomposed `écase`, `℘case`,
+   trailing-mark spoofs without a full Unicode-identifier lexer (Codex finding, folded).
+3. **WHEN-lookahead gate** (`_case_has_when`): scan after CASE at relative depth 0
+   (parens + brackets); FIRST structural keyword decides — `WHEN` → conditional, any of
+   `THEN`/`ELSE`/`END`/`CASE` → identifier; unmatched `)` or end → identifier.
+   Alternative rejected (interview Q1, upheld against a Codex objection): backtracking
+   parse that would also accept a bare keyword-named simple-CASE operand
+   (`CASE case WHEN 1 …`) — real grammar ambiguity, disproportionate complexity; the
+   parenthesized operand `CASE (case) WHEN …` works under the gate and is documented.
+4. **ASCII keyword classes instead of `re.IGNORECASE`** for `_SQL_LIKE_RE` (`like`,
+   `not`), `_OVER_RE`, and the keyword rewrites in `_normalize_sql_filter_operators`:
+   IGNORECASE folds `lıke` (dotless ı) and `liKe` (KELVIN) into `like` (verified).
+   `[lL][iI][kK][eE]`-style classes keep every ASCII casing and nothing else.
+5. **LIKE literal-span guard**: precompute string-literal spans with
+   `_PY_STRING_LITERAL_RE`; a match starting inside a span is returned unchanged. RHS
+   group becomes the escape-aware `'(?:\\.|[^'\\])*'`. Single-quote-only by decision
+   (interview Q3): pg-facade SQL double quotes denote identifiers.
+6. **`parse_filter` retirement**: `_filter_refs_dsl` reimplemented as
+   `parse_filter_expr` + the node-walk already used by `_measure_formula_refs`
+   (shared helper extracted; DEV-1826 expression-source descent included). Reference
+   order changes from measures-first to expression order — authorized: the sole caller
+   (`_filter_refs_on_base`) aggregates into a set. Then delete `parse_filter` and every
+   private helper reaching zero references, iterating to fixpoint:
+   `_preprocess_like`, `_LIKE_RE`, `_preprocess_sql_operators`, `_preprocess_concat`,
+   `_filter_node_to_sql`, `_call_to_sql`, `_compare_to_sql`, `_binop_to_sql`,
+   `_flatten_lshift_chain`, `_LIKE_INTERNAL_NAMES` + the `__like__` branch of
+   `_classify_call_name`; audit `AggRef` / `_SUBQUERY_IN_FILTER_RE` for other users
+   before deleting. Retained: `parse_formula` (converters), `_preprocess_agg_refs`,
+   `_rewrite_funcstyle_aggregations`, `has_window_function`, shared constants.
+7. **`ParsedFilter` moves to `slayer/sql/sql_predicate.py`** trimmed to `sql` +
+   `columns` — its only remaining producer is `parse_sql_predicate`; `agg_refs` /
+   `synthesized_aliases` / `is_having` / `is_post_filter` have zero remaining readers.
+8. **Stale-comment scrub**: syntax.py's "module DEV-1452 deletes" claim (DEV-1452
+   completed and deliberately retained formula.py) and the `_preprocess_like` mirror
+   claims; `parse_filter` mentions in docs/architecture.
+
+## Risks / Trade-offs
+
+- [Keyword-named bare simple-CASE operand and nested-CASE-as-operand mislex] → loud
+  parse/plan error, never silent corruption; parenthesize escape hatch documented in
+  code comment; spec scenario pins the behavior.
+- [Legacy test semantics lost in migration] → DEV-1452 Stage C pattern: audit each of
+  the ~52 call sites; delete only what typed-parser tests already cover; migrate the
+  rest onto `parse_filter_expr`.
+- [Drift-refs parity regression] → dedicated parity suite (colon/funcstyle/custom aggs,
+  dotted paths, expression sources, `*` exclusion, failure → `[]`) written before the
+  migration lands.
+
+## Migration Plan
+
+Single PR, sequenced commits: (1) hardening + new regression tests, (2)
+`_filter_refs_dsl` migration + parity tests, (3) deletions + test migration + comment
+scrub. Rollback = revert; no storage or API surface touched.

openspec/changes/archive/2026-09-04-dev-1833-harden-mode-b-keyword-lexing-case-like-for-unicode/proposal.md

@@ -0,0 +1,63 @@
+# Proposal: Harden Mode-B keyword lexing (CASE / LIKE) for Unicode identifiers
+
+## Why
+
+The Mode-B pre-`ast.parse` textual preprocessors in `slayer/engine/syntax.py` key off SQL
+keywords with ASCII-oriented regex patterns, so a legal identifier can be misread as a
+keyword: a column named `case`, a qualified `customers.case`, or a Unicode-prefixed
+`écase` all fail to parse today, and `customers.end` inside a THEN value silently
+corrupts the rewritten expression. The sibling LIKE rewriter corrupts expressions whose
+string literals contain ` like ` and patterns with escaped quotes. Same bug class as the
+`__slayer_` boundary fix in DEV-1743 / PR #334.
+
+Additionally (interview-approved scope extension): the legacy `formula.parse_filter` is a
+second, near-dead Mode-B filter parser — its only production caller is
+`schema_drift._filter_refs_dsl`, and its `__like__`/`__notlike__` machinery exists solely
+to feed itself. Retiring it leaves exactly one Mode-B filter parser and one LIKE rewriter.
+
+## What Changes
+
+- `_CASE_TOKEN_RE` lexes complete identifiers: Unicode-aware start, dotted paths (with
+  optional whitespace around dots) as single tokens.
+- Keyword recognition requires ASCII tokens (blocks `'caſe'.upper() == 'CASE'` spoofs)
+  and rejects tokens adjacent to identifier-material characters the regex `\w` class
+  misses (combining marks, `Other_ID_Start` symbols like `℘`).
+- `_rw_case` is entered only when a depth-0 `WHEN` follows the `CASE` token; otherwise
+  `case` flows through as an ordinary identifier.
+- `_SQL_LIKE_RE`: keyword matched as explicit ASCII character classes (drops
+  `re.IGNORECASE`, which folds `lıke`/`liKe` into `like`); escape-aware pattern
+  literal; matches starting inside string literals are skipped. Same ASCII-classes
+  treatment for `_OVER_RE` and the keyword rewrites in `_normalize_sql_filter_operators`.
+- **BREAKING (internal only)**: legacy `formula.parse_filter` and its private subtree
+  (`_preprocess_like`, `_preprocess_sql_operators`, the `__like__`/`__notlike__` helpers,
+  the Mode-B filter→SQL lowering functions) are deleted; `schema_drift._filter_refs_dsl`
+  moves onto the typed `parse_filter_expr`; trimmed `ParsedFilter` (`sql` + `columns`)
+  moves to `slayer/sql/sql_predicate.py`. No public API changes.
+- Error-surface change: a bare `CASE`-named reference no longer raises "Malformed CASE";
+  a `CASE` with no `WHEN` at all degrades to the generic invalid-expression error.
+
+## Capabilities
+
+### New Capabilities
+
+- `queries/expression-keywords`: SQL keyword affordances inside Mode-B expressions —
+  CASE WHEN lowering to `iif`, LIKE/NOT LIKE rewriting to the `like()` scalar — and the
+  identifier-safety rules guaranteeing that identifiers named after, containing, or
+  qualified by SQL keywords are never captured by that recognition.
+
+### Modified Capabilities
+
+(none — the `parse_filter` retirement preserves the drift-refs contract; no existing
+spec's requirements change)
+
+## Impact
+
+- `slayer/engine/syntax.py` — tokenizer, CASE gate, LIKE/OVER/operator keyword regexes.
+- `slayer/core/formula.py` — `parse_filter` subtree deleted (~several hundred LOC);
+  `parse_formula`, `_preprocess_agg_refs`, `_rewrite_funcstyle_aggregations`, constants retained.
+- `slayer/engine/schema_drift.py` — `_filter_refs_dsl` on the typed parser; reference
+  order becomes expression order (verified: sole caller aggregates into a set).
+- `slayer/sql/sql_predicate.py` — receives the trimmed `ParsedFilter`.
+- Tests: new regression file; ~52 legacy `parse_filter` call sites across 4 files
+  migrated or deleted (DEV-1452 Stage C pattern, user-consented).
+- Docs: one sentence in `docs/concepts/formulas.md`.

openspec/changes/archive/2026-09-04-dev-1833-harden-mode-b-keyword-lexing-case-like-for-unicode/specs/queries/expression-keywords/spec.md

@@ -0,0 +1,131 @@
+## Purpose
+
+SQL keyword affordances inside Mode-B expressions — `CASE WHEN` lowering to `iif` and
+`LIKE`/`NOT LIKE` rewriting to the `like()` scalar — and the identifier-safety rules
+guaranteeing that identifiers named after, containing, or qualified by SQL keywords are
+never captured by that recognition.
+
+## ADDED Requirements
+
+### Requirement: CASE WHEN lowering
+
+A Mode-B expression SHALL accept SQL `CASE … END` conditionals — searched
+(`CASE WHEN cond THEN val … [ELSE val] END`) and simple
+(`CASE operand WHEN val THEN val … [ELSE val] END`) — lowering them to nested
+`iif(cond, then, otherwise)` calls with `None` as the default otherwise. WHEN
+conditions SHALL accept SQL operator spellings (`=`, `<>`, `AND`/`OR`/`NOT`,
+`IS [NOT] NULL`, `[NOT] IN`, `[NOT] LIKE`) in every expression position, including
+measures. A `CASE` that is recognized as a conditional (a `WHEN` follows it) but is
+malformed MUST raise a specific malformed-CASE error naming the defect.
+
+#### Scenario: searched CASE lowers to iif
+
+- WHEN `CASE WHEN amount > 100 THEN 'big' ELSE 'small' END` is parsed as a Mode-B expression
+- THEN it parses as `iif(amount > 100, 'big', 'small')` — a scalar call, usable wherever a scalar expression is legal
+
+#### Scenario: simple CASE compares the operand per branch
+
+- WHEN `CASE status WHEN 'a' THEN 1 WHEN 'b' THEN 2 END` is parsed
+- THEN it parses as `iif(status == 'a', 1, iif(status == 'b', 2, None))`
+
+#### Scenario: nested CASE in THEN and ELSE values
+
+- WHEN a THEN or ELSE value itself contains a complete `CASE … END`
+- THEN the nested conditional is lowered recursively and the enclosing branches are unaffected
+
+#### Scenario: SQL operator spellings inside WHEN conditions
+
+- WHEN `CASE WHEN region = 'EU' AND amount IS NOT NULL THEN 1 ELSE 0 END` appears in a measure formula
+- THEN the WHEN condition is normalized (`==`, `and`, `is not None`) and the expression parses
+
+#### Scenario: recognized-but-malformed CASE still errors specifically
+
+- WHEN `CASE WHEN a THEN 1` (missing END) or `CASE WHEN a 1 END` (missing THEN) is parsed
+- THEN a malformed-CASE error is raised naming the missing keyword, not a generic syntax error
+
+### Requirement: keyword-named identifiers are never captured
+
+An identifier that is merely named after, prefixed by, containing, or qualified by a SQL
+keyword SHALL parse as an ordinary reference in every Mode-B expression position. `CASE`
+SHALL be treated as a conditional only when a `WHEN` token follows it at parenthesis
+depth 0 before any other structural keyword (`THEN`/`ELSE`/`END`/`CASE`), an unmatched
+closing parenthesis, or end of input. Keyword recognition MUST be ASCII-exact: tokens
+whose uppercase form only coincides with a keyword via Unicode case folding, and tokens
+adjacent to identifier-forming characters outside the regex word class (combining marks,
+`Other_ID_Start` symbols), are ordinary identifiers. Dotted references qualify their
+leaf regardless of whitespace around the dots.
+
+#### Scenario: bare keyword-named column
+
+- WHEN `case` (or `case + 1`, or `iif(case, 1, 2)`) is parsed as a Mode-B expression
+- THEN `case` resolves as an ordinary column reference and no CASE lowering occurs
+
+#### Scenario: qualified keyword-named column
+
+- WHEN `customers.case` is parsed, with or without whitespace around the dot (`customers . case`)
+- THEN it parses as a dotted reference to the `case` column of `customers`
+
+#### Scenario: Unicode identifiers containing keywords
+
+- WHEN `écase`, `变量`, decomposed `écase`, or `℘case` is parsed
+- THEN each parses as a single ordinary identifier; no fragment of it is read as a keyword
+
+#### Scenario: Unicode case-fold spoofs are not keywords
+
+- WHEN an identifier like `caſe` (uppercases to `CASE`) appears in an expression
+- THEN it is an ordinary identifier, not a CASE keyword
+
+#### Scenario: keyword-named identifier alongside a real CASE
+
+- WHEN `case + CASE WHEN x THEN 1 END` or `CASE WHEN case THEN 1 WHEN other THEN 2 END` is parsed
+- THEN the bare `case` references stay identifiers while the real `CASE WHEN … END` lowers to `iif`
+
+#### Scenario: keyword-named dotted reference inside CASE branch values
+
+- WHEN `CASE WHEN a THEN customers.end ELSE 0 END` is parsed
+- THEN the THEN value is the complete `customers.end` reference and the conditional lowers correctly
+
+#### Scenario: bare CASE with no WHEN is not a conditional
+
+- WHEN `CASE` appears with no depth-0 `WHEN` following (e.g. the whole expression is `case` or `case_total * 2`)
+- THEN no CASE lowering is attempted; the text parses (or fails) exactly as if `case` were any other identifier
+
+#### Scenario: keyword-named simple-CASE operand requires parentheses
+
+- WHEN `CASE case WHEN 1 THEN 2 END` is parsed
+- THEN the ambiguous bare keyword-named operand raises an error (never silent corruption), and the parenthesized form `CASE (case) WHEN 1 THEN 2 END` parses correctly with `case` as the operand reference
+
+### Requirement: LIKE operator rewriting
+
+A Mode-B filter SHALL accept `lhs [NOT] LIKE 'pattern'` — LHS a bare or dotted
+identifier or single scalar call, pattern a single-quoted string literal with
+backslash-escape support — rewriting it to the `like(lhs, pattern)` scalar (negated:
+`not like(...)`). The keyword match SHALL be ASCII-exact (any ASCII casing; never via
+Unicode case folding) and SHALL never apply inside a string literal. A double-quoted
+pattern is NOT rewritten (in SQL sources double quotes denote identifiers), so it fails
+loudly rather than silently changing meaning.
+
+#### Scenario: basic LIKE and NOT LIKE
+
+- WHEN `name LIKE 'a%'` / `name NOT LIKE 'a%'` / `lower(customers.email) like '%@x.io'` appear in a filter
+- THEN each rewrites to the corresponding `like(...)` / `not like(...)` scalar call
+
+#### Scenario: escaped quote inside the pattern
+
+- WHEN `col LIKE 'It\'s%'` appears in a filter
+- THEN the full pattern including the escaped quote is preserved as the second argument
+
+#### Scenario: LIKE inside a string literal is untouched
+
+- WHEN a filter contains ` like ` only inside a string literal, e.g. `note == "we like 'cats'"`
+- THEN the literal is preserved byte-for-byte and no rewrite occurs
+
+#### Scenario: case-fold keyword spoofs are not LIKE
+
+- WHEN a filter contains `x lıke 'p%'` or `x liKe 'p%'` (dotless ı / KELVIN SIGN fold to `like`)
+- THEN no rewrite occurs (the token is an ordinary identifier), while ASCII `x LiKe 'p%'` still rewrites
+
+#### Scenario: double-quoted pattern is rejected loudly
+
+- WHEN `col like "p%"` appears in a filter
+- THEN parsing fails with an invalid-expression error rather than rewriting to a string match