Reject a trailing operator instead of dropping it at end of input - #306
Merged
Conversation
isEOF only reports that no input text is left to lex, which is already true while the final token sits unconsumed in the lookahead. The infix loop stopped on it, so an operator at the end of the input was dropped rather than reported as missing its right operand. Most such inputs still failed, but on the wrong token: SELECT a + left the plus for the statement parser to reject as unexpected trailing input. Two were accepted outright, because a dropped operator keyword fell through to the implicit-alias logic: SELECT a GLOBAL parsed as SELECT a AS GLOBAL, and SELECT a REGEXP as SELECT a AS REGEXP. The invalid-syntax tests already record that GLOBAL is never an implicit alias; it only slipped through at end of input. Guard the loop on the lookahead token instead, which is the idiom parseColumnExprListWithTerm and parseSelectItems already use. Entering parseInfix on the final token is safe: every branch reads the current token behind a match check, and each path to a right operand is nil-safe at EOF. No valid statement ends in an infix operator, so the change only turns silently truncated parses into errors, and no golden file moves. clickhouse-local 26.7.1 rejects all four new cases, and accepts the explicit SELECT 1 AS GLOBAL that the alias reading was confused with. Assistant By Claude Opus 5
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
pull Bot
pushed a commit
to edisplay/signoz
that referenced
this pull request
Aug 7, 2026
Bumps `clickhouse-sql-parser` to v0.5.5, fixes the false rejection that was left over once it landed, and closes three holes in the same validator that the first two changes brought to light. ## The bump **Reserved keywords as expression operands** ([#305](AfterShip/clickhouse-sql-parser#305)). `interval` was fixed in v0.5.4, but the same defect affected 36 other keywords once the column appeared as an operand rather than bare. Sweeping 94 candidates against ClickHouse 26.8.1.337, only `on` still rejects — and ClickHouse runs that too. This one was live: `sum(limit)` on a metric label. **Panic on an unparseable `DEFAULT` expression** ([#306](AfterShip/clickhouse-sql-parser#306)). Both known cases return a parse error now instead of dereferencing nil. The `recover` in `ErrIfStatementIsNotValid` stays — it guards the next one of these, not these two. [#307](AfterShip/clickhouse-sql-parser#307) also allows `CAST` in a table function's argument list. ## Table functions are only table functions in a table position The parser types a call inside a table function's argument list as a `TableFunctionExpr` as well, so the generator allow list only ever cleared a generator whose argument was a literal. Every real dashboard computes its row count — `numbers(greatest(1, intDiv(end_ns - start_ns, step_ns) + 1))` — and every one was refused, on `intDiv` rather than on `numbers`. `TableExpr.Expr` is the only table position a SELECT can reach, so the allow list asks that instead. Of the four places the parser builds a `TableFunctionExpr`, two are `CREATE TABLE` paths rejected as not-a-SELECT before the walk starts, one is `parseTableArgPrimaryExpr`, and one is the `FROM`/`JOIN` path that wraps into a `TableExpr`. ## Three holes that were already open Skipping argument position is only safe if nothing there can read, and that turned out not to be true — not because of this change, but independently of it. **Reading functions.** `file` is both a table function and a scalar function, and the validator never inspected scalar calls at all. On `main` today, `SELECT file('/etc/passwd')` is accepted and returns the file. A numeric wrapper passes ClickHouse's type check, so the row count alone is an oracle: `numbers(length(file(x)))` yields one row per byte. The same applies to the 42 dictionary accessors, which can be backed by HTTP, ODBC or another database, to `catboostEvaluate`, and to the introspection functions. All are now refused by name wherever they appear, under `clickhouse_sql_reading_function`. **`x IN db.table`.** ClickHouse reads this as `x IN (SELECT * FROM db.table)`, and a qualified name on the right of `IN` parses as a `Path`, not a `TableIdentifier` — so `SELECT * FROM t WHERE a IN system.users` bypassed the internal-database rule entirely. Now checked, including the `GLOBAL IN` and `NOT IN` forms. **Quoted generator names.** The allow list matched on the formatted name, which carries the quoting, so ``SELECT * FROM `numbers`(31)`` was refused. It now reads the identifier the way the internal-database branch already did. ## Effect Replaying 72 distinct shapes of production `clickhouse_sql` that the validator currently rejects: **64 pass, up from 59 on v0.5.4**. Two came from the bump, three from the table-position change, and those three are 379 of the 1390 sampled occurrences. The three new rules add no false positives to the corpus. Of the eight left, four are correct rejections (`system` reads, `SHOW TABLES`), one is a dashboard variable rendering as the literal `<no value>`, one is SQL ClickHouse also rejects, and two are an open upstream gap. ## Tests `TestErrIfStatementIsNotValid_ShouldPassButFails` is back, holding what remains: three forms of a parenthesised left operand of a set operator, and `on` as a column name. It also stopped panicking — `errors.Asc` dereferences the error it is given, so a case starting to pass took the suite out with a SIGSEGV instead of reporting. Both refusal tables now share one harness, bounded by the same timeout the passing table uses. Known gap: no input is currently known to panic the parser, so the `recover` has no test exercising it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
isEOF only reports that no input text is left to lex, which is already
true while the final token sits unconsumed in the lookahead. The infix
loop stopped on it, so an operator at the end of the input was dropped
rather than reported as missing its right operand.
Most such inputs still failed, but on the wrong token: SELECT a + left
the plus for the statement parser to reject as unexpected trailing
input. Two were accepted outright, because a dropped operator keyword
fell through to the implicit-alias logic: SELECT a GLOBAL parsed as
SELECT a AS GLOBAL, and SELECT a REGEXP as SELECT a AS REGEXP. The
invalid-syntax tests already record that GLOBAL is never an implicit
alias; it only slipped through at end of input.
Guard the loop on the lookahead token instead, which is the idiom
parseColumnExprListWithTerm and parseSelectItems already use. Entering
parseInfix on the final token is safe: every branch reads the current
token behind a match check, and each path to a right operand is nil-safe
at EOF. No valid statement ends in an infix operator, so the change only
turns silently truncated parses into errors, and no golden file moves.
clickhouse-local 26.7.1 rejects all four new cases, and accepts the
explicit SELECT 1 AS GLOBAL that the alias reading was confused with.
🤖 Generated with Claude Code