Skip to content

fix(substrait): roundtrip scalar subqueries in projections#23469

Draft
bvolpato wants to merge 1 commit into
apache:mainfrom
bvolpato:agent/substrait-scalar-subquery-projection
Draft

fix(substrait): roundtrip scalar subqueries in projections#23469
bvolpato wants to merge 1 commit into
apache:mainfrom
bvolpato:agent/substrait-scalar-subquery-projection

Conversation

@bvolpato

@bvolpato bvolpato commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

With datafusion.optimizer.enable_physical_uncorrelated_scalar_subquery disabled, DataFusion rewrites a scalar subquery in a projection as a left join. The optimizer can remove its true filter before Substrait serialization. This left JoinRel.expression unset, even though Substrait requires it, and the consumer failed with Plan("join condition should not be empty").

#18068 tried to fix the same roundtrip in the consumer, but also treated a conditionless inner JoinRel as a cross join. Review questioned that behavior change, then the PR went stale. This version does not reinterpret an invalid JoinRel: the producer uses CrossRel for conditionless inner joins and a literal true expression for conditionless non-inner joins.

What changes are included in this PR?

  • Serialize conditionless inner joins as CrossRel.
  • Serialize a literal true expression for conditionless non-inner joins.
  • Add a scalar-subquery projection roundtrip test that checks the consumed plan and query result.

Are these changes tested?

  • cargo test -p datafusion-substrait --features protoc
  • cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

Scalar subqueries in projections now roundtrip when this optimizer option is disabled. No public API change.

@github-actions github-actions Bot added the substrait Changes to the substrait crate label Jul 10, 2026
@bvolpato bvolpato force-pushed the agent/substrait-scalar-subquery-projection branch from 3537b75 to 0f01204 Compare July 11, 2026 21:01
@bvolpato bvolpato force-pushed the agent/substrait-scalar-subquery-projection branch from 0f01204 to 35f7501 Compare July 11, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[substrait] Scalar subquery in select not supported

1 participant