Repository navigation
PR #62 follow-up 19: run tool SQL ending in '; -- comment' - #74
Merged
Merged
Conversation
bounded_tool_sql removed the terminator with trim_end_matches(';'), which
left the ';' inside the LIMIT wrapper whenever a comment followed it, so
`SELECT a FROM t; -- note` passed validation and then failed to parse.
Cut the query at the first ';' of the trailing run of ';'/comment tokens
instead. Document that projection limits apply only to the outer query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix/pr62-19-sql-wrapper-projection # Conflicts: # docs/tool-api.md
… fix/pr62-19-sql-wrapper-projection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Originally stacked on #62. #62 has merged, and this PR now targets
main.Fixes review follow-up 19 from #62. The code is upstream-inherited, from parseablehq/parseable 32979cb.
Problem
bounded_tool_sqlwraps the caller's query asSELECT * FROM (<query>\n) AS "__parseable_tool_result" LIMIT 500. It removes the statement terminator withquery.trim().trim_end_matches(';'), which only works when the text ends in;. DataFusion's parser acceptsSELECT a FROM t; -- noteas one statement, so validation passes. But the;is not the last character, so it stays inside the wrapper:So
query_sqlandexplain_queryreturn a confusing runtime parse error for a query the validator accepted. The same happens with; /* note */and; ;.Fix
A new helper,
strip_trailing_semicolons, tokenizes the query with sqlparser's tokenizer, usingGenericDialect, the same dialectDFParseruses. It finds the trailing run of;and whitespace or comment tokens and cuts the original text at the first;in that run. Otherwise the caller's SQL is passed through verbatim. A;inside a string literal or a comment is never treated as the terminator. A trailing comment with no semicolon is still preserved, as the existing upstream test expects.I didn't take the alternative of rebuilding the SQL from the AST (
statement.to_string()). It re-renders the whole query, so formatting, quoting and literal spelling can drift from what the caller wrote and what DataFusion would plan. Cutting at a token boundary keeps the change minimal.Projection checks inside CTEs and subqueries (optional part, not done)
validate_projectionstill checks only the outermost query body: parenthesized queries and each branch of a set operation. It does not descend into CTEs, derived tables or subquery expressions. Extending it would reject legitimate SQL such asWHERE EXISTS (SELECT * …)andWITH x AS (SELECT * FROM t) SELECT a, b FROM x. Such queries can come from agents reusing dashboard tile SQL. Meanwhile, the result width is already bounded by the outer projection check: at most 10 explicit items and no*, in every branch of a set operation. The fork's generated queries (list_log_errors_query,trace_activity_query,sample_events_query) are flat and unaffected either way. New assertions pin down that behaviour, whichdocs/tool-api.mdon #62 already describes accurately.Docs
#62's "SQL tool limits" section documented the semicolon-plus-comment failure as a known limitation. This PR replaces that sentence with the fixed behaviour: trailing semicolons and comments are allowed, including comments after the final semicolon.
Tests
bounded_tool_sql_runs_with_comments_after_the_trailing_semicolonexecutes the wrapped SQL in a real DataFusionSessionContextfor eight cases:; -- c,; /* c */,; ;followed by comments, a comment before;,';'inside a string literal,;inside a line comment,;inside a block comment, and a non-ASCII literal (column-to-byte offset mapping). Fails without the fix with theParserErrorabove.shared_sql_limits_reject_wildcards_and_excess_columnsnow asserts that*inside a CTE, derived table orEXISTSis accepted and that an outerSELECT *over a CTE is rejected.cargo test --lib tool_passed, on both the branch and the merge with the latest base.cargo clippy -- -D warningsis clean. No new--all-targetswarnings in touched files.Upstream fidelity
The change touches upstream-owned code in
src/tool_executor.rs: one added helper, one changed line inbounded_tool_sql, two new imports, and tests. It's small and self-contained, so future upstream merges should conflict only if upstream rewritesbounded_tool_sqlitself. The same fix is worth proposing upstream.🤖 Generated with Claude Code