[SPARK-59371][SQL] Gate the startsAndEndsWith LikeSimplification rewrite on a cheap child - #58663
Conversation
…ite on a cheap child `LikeSimplification` rewrites `LIKE 'prefix%suffix'` (the `startsAndEndsWith` shape, e.g. `'a%b'`) into `lengthGuard(input) && StartsWith(input, prefix) && EndsWith(input, postfix)`, referencing the child three times. The single-`Like` branch of `LikeSimplification.apply` applied this unconditionally, unlike the multi-LIKE branches, which only fire when `CollapseProject.isCheap(child)` (SPARK-40228). For a non-cheap child this duplicated the child: a nondeterministic child (e.g. `uuid()`) was evaluated three times with different values -- changing results -- and an expensive child (e.g. `sha2`) was evaluated repeatedly. Gate the `startsAndEndsWith` case on `CollapseProject.isCheap(input)`, leaving the pattern as a plain `Like` for a non-cheap child. The single-reference shapes evaluate the child once, like the original `LIKE`, so they remain enabled for any child. Generated-by: Claude Opus 4.8 Co-authored-by: Isaac <no-reply@databricks.com>
|
The failing
I've opened #58712 (SPARK-59413) to fix that at the source: teach #58712 is a prerequisite for this PR. Once it merges and this branch is rebased on |
What changes were proposed in this pull request?
LikeSimplificationrewrites aLIKE 'prefix%suffix'pattern (thestartsAndEndsWithshape, e.g.
'a%b') intowhich references the child
inputthree times. The single-Likebranch ofLikeSimplification.applyapplied this rewrite unconditionally, unlike theLikeAll/NotLikeAll/LikeAny/NotLikeAnybranches, which only fire whenCollapseProject.isCheap(child)(SPARK-40228).This PR gates the
startsAndEndsWithcase onCollapseProject.isCheap(input). For anon-cheap child the pattern is left as a plain
Like. The single-reference shapes(
startsWith,endsWith,contains,equalTo) reference the child once and remainenabled for any child.
Why are the changes needed?
Duplicating a non-cheap child is both a correctness and a performance problem:
uuid(),cast(rand() as string)) isevaluated independently for each reference, so the three copies can produce different
values and the rewritten predicate no longer matches the semantics of the original
LIKE. Subexpression elimination does not help: it deliberately never deduplicatesnondeterministic expressions.
sha2(col)) is written intothe plan three times; the logical plan should not rely on subexpression elimination
collapsing the repeats during codegen.
This is the same duplication class that SPARK-40228 fixed for the multi-
LIKErules.Does this PR introduce any user-facing change?
Yes. For a
LIKE 'prefix%suffix'pattern over a nondeterministic child, the child is nowevaluated once instead of once per reference. For example,
uuid() LIKE 'a%b':length(uuid()) >= 2 AND startswith(uuid(), 'a') AND endswith(uuid(), 'b'), drawingthree independent UUIDs and testing the length, prefix, and suffix of different strings.
uuid() LIKE 'a%b', drawing a single UUID and matching it againstthe whole pattern.
The new behavior matches evaluating the
LIKEdirectly (a single evaluation of thechild), which is what users expect. Queries with a nondeterministic argument to such a
LIKEcan return different rows than before. Because thestartsAndEndsWithrewrite islong-standing, this is a user-facing change relative to released Spark versions as well as
master. Behavior is unchanged for cheap children (attributes, foldables, etc.), thecommon case.
How was this patch tested?
Added two unit tests to
LikeSimplificationSuite:SPARK-59371: do not simplify startsAndEndsWith LIKE for a non-cheap child— asserts$"a".substring(1, 5) like "a%b"is left asLike(fails before this change).SPARK-59371: still simplify single-reference LIKE shapes for a non-cheap child—asserts the
startsWith/endsWith/contains/equalToshapes still simplify for anon-cheap child (guards against over-gating).
build/sbt 'catalyst/testOnly *LikeSimplificationSuite'passes (22 tests); scalastyleclean.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
This pull request and its description were written by Isaac.