Skip to content

[SPARK-59413][SQL] Treat the Collate passthrough as cheap in CollapseProject.isCheap - #58712

Open
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:ischeap-collate-passthrough
Open

[SPARK-59413][SQL] Treat the Collate passthrough as cheap in CollapseProject.isCheap#58712
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:ischeap-collate-passthrough

Conversation

@david-mollitor-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

CollapseProject.isCheap(e: Expression) decides whether an expression is cheap enough to
duplicate/inline. It recognizes Attribute / OuterReference / BoundReference, foldable
expressions, some PythonUDFs, and Alias / ExtractValue over cheap children — but not the
Collate expression. This PR teaches it that Collate is as cheap as its value child:

// `Collate` only re-tags the collation in the type; at runtime it is a pass-through
// (eval/genCode delegate to the child) and never evaluates its collation argument, so it is
// as cheap to duplicate as its value child.
case c: Collate => isCheap(c.child)

Only the value child is inspected — the collation argument is a constant ResolvedCollation
marker that is never evaluated — so a non-cheap value child (e.g. Collate(substring(col), ...))
correctly stays non-cheap, preserving the SPARK-40228 guard.

Why are the changes needed?

Collate (the SQL collate(expr, 'COLLATION') function / expr COLLATE COLLATION syntax) is a
pure pass-through: its eval delegates to the child (child.eval(row)), its codegen delegates to
the child (child.genCode(ctx)), and it never evaluates its collation argument. It performs no
per-row work — it only re-tags the collation carried on the result type. This is stated in the
expression's own doc comment:
Collate is pass-through.

Because isCheap did not know this, Collate(cheapChild) was treated as non-cheap, needlessly
blocking safe duplication/inlining of collated columns across every isCheap-gated rule
(CollapseProject inlining, the multi-LIKE LikeSimplification rules, the FilterExec CSE gate,
RewriteWithExpression).

Does this PR introduce any user-facing change?

No. Collate returns the child's value unchanged and only sets collation metadata on the type, so
duplicating Collate(cheapChild) is semantically identical to duplicating the child. Results are
unchanged for all inputs; the optimizer may now inline/duplicate a collated column where it
previously could not.

How was this patch tested?

Added a CollapseProjectSuite test asserting isCheap(Collate(attr, ...)) is true while
isCheap(Collate(substring(attr), ...)) is false (the SPARK-40228 protection stays intact).
Existing CollapseProjectSuite and LikeSimplificationSuite suites pass, and scalastyle is
clean.

This also unblocks the collated-column case of the gated startsAndEndsWith LikeSimplification
rewrite in SPARK-59371: with Collate
recognized as cheap, collate(col) LIKE 'a%c' is simplified again instead of being left as a plain
Like.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

This pull request and its description were written by Isaac.

…Project.isCheap

`CollapseProject.isCheap` decides whether an expression is cheap enough to
duplicate/inline. It did not recognize the `Collate` expression, so
`Collate(cheapChild)` was treated as non-cheap and blocked from safe
duplication/inlining across every `isCheap`-gated rule (CollapseProject inlining,
the multi-LIKE `LikeSimplification` rules, the `FilterExec` CSE gate,
`RewriteWithExpression`).

`Collate` (the SQL `collate(expr, 'COLLATION')` function / `expr COLLATE COLLATION`
syntax) is a pure pass-through: its `eval` and codegen delegate directly to the
child and it never evaluates its collation argument -- it only re-tags the collation
carried on the result type. It is deterministic and, at runtime, exactly as cheap to
duplicate as the child it wraps.

Teach `isCheap` to look through `Collate` to its value child. Only the value child is
inspected -- the collation argument is a constant `ResolvedCollation` marker that is
never evaluated -- so a non-cheap value child (e.g. `Collate(substring(col), ...)`)
correctly stays non-cheap, preserving the SPARK-40228 guard.

This is behavior-preserving: `Collate` returns the child's value unchanged, so
duplicating `Collate(cheapChild)` is semantically identical to duplicating the child.
It unblocks the collated-column case of the gated `startsAndEndsWith`
`LikeSimplification` rewrite (SPARK-59371).

Generated-by: Claude Opus 4.8
Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant