[SPARK-59413][SQL] Treat the Collate passthrough as cheap in CollapseProject.isCheap - #58712
Open
david-mollitor-db wants to merge 1 commit into
Open
[SPARK-59413][SQL] Treat the Collate passthrough as cheap in CollapseProject.isCheap#58712david-mollitor-db wants to merge 1 commit into
david-mollitor-db wants to merge 1 commit into
Conversation
…Project.isCheap `CollapseProject.isCheap` decides whether an expression is cheap enough to duplicate/inline. It did not recognize the `Collate` expression, so `Collate(cheapChild)` was treated as non-cheap and blocked from safe duplication/inlining across every `isCheap`-gated rule (CollapseProject inlining, the multi-LIKE `LikeSimplification` rules, the `FilterExec` CSE gate, `RewriteWithExpression`). `Collate` (the SQL `collate(expr, 'COLLATION')` function / `expr COLLATE COLLATION` syntax) is a pure pass-through: its `eval` and codegen delegate directly to the child and it never evaluates its collation argument -- it only re-tags the collation carried on the result type. It is deterministic and, at runtime, exactly as cheap to duplicate as the child it wraps. Teach `isCheap` to look through `Collate` to its value child. Only the value child is inspected -- the collation argument is a constant `ResolvedCollation` marker that is never evaluated -- so a non-cheap value child (e.g. `Collate(substring(col), ...)`) correctly stays non-cheap, preserving the SPARK-40228 guard. This is behavior-preserving: `Collate` returns the child's value unchanged, so duplicating `Collate(cheapChild)` is semantically identical to duplicating the child. It unblocks the collated-column case of the gated `startsAndEndsWith` `LikeSimplification` rewrite (SPARK-59371). Generated-by: Claude Opus 4.8 Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
CollapseProject.isCheap(e: Expression)decides whether an expression is cheap enough toduplicate/inline. It recognizes
Attribute/OuterReference/BoundReference, foldableexpressions, some
PythonUDFs, andAlias/ExtractValueover cheap children — but not theCollateexpression. This PR teaches it thatCollateis as cheap as its value child:Only the value child is inspected — the collation argument is a constant
ResolvedCollationmarker that is never evaluated — so a non-cheap value child (e.g.
Collate(substring(col), ...))correctly stays non-cheap, preserving the SPARK-40228 guard.
Why are the changes needed?
Collate(the SQLcollate(expr, 'COLLATION')function /expr COLLATE COLLATIONsyntax) is apure pass-through: its
evaldelegates to the child (child.eval(row)), its codegen delegates tothe child (
child.genCode(ctx)), and it never evaluates its collation argument. It performs noper-row work — it only re-tags the collation carried on the result type. This is stated in the
expression's own doc comment:
Collate is pass-through.
Because
isCheapdid not know this,Collate(cheapChild)was treated as non-cheap, needlesslyblocking safe duplication/inlining of collated columns across every
isCheap-gated rule(CollapseProject inlining, the multi-LIKE
LikeSimplificationrules, theFilterExecCSE gate,RewriteWithExpression).Does this PR introduce any user-facing change?
No.
Collatereturns the child's value unchanged and only sets collation metadata on the type, soduplicating
Collate(cheapChild)is semantically identical to duplicating the child. Results areunchanged for all inputs; the optimizer may now inline/duplicate a collated column where it
previously could not.
How was this patch tested?
Added a
CollapseProjectSuitetest assertingisCheap(Collate(attr, ...))istruewhileisCheap(Collate(substring(attr), ...))isfalse(the SPARK-40228 protection stays intact).Existing
CollapseProjectSuiteandLikeSimplificationSuitesuites pass, andscalastyleisclean.
This also unblocks the collated-column case of the gated
startsAndEndsWithLikeSimplificationrewrite in SPARK-59371: with
Collaterecognized as cheap,
collate(col) LIKE 'a%c'is simplified again instead of being left as a plainLike.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.