Skip to content

Do not broadcast when co-partitioned. - #25304

Open
stuhood wants to merge 1 commit into
apache:mainfrom
paradedb:stuhood.do-not-broadcast-when-copartitioned
Open

stuhood wants to merge 1 commit into
apache:mainfrom
paradedb:stuhood.do-not-broadcast-when-copartitioned

Conversation

@stuhood

@stuhood stuhood commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

In JoinSelection, try_collect_left converts a hash join to PartitionMode::CollectLeft (broadcast) when either input is below configured byte/row thresholds. However, if the inputs are already co-partitioned on the join keys across multiple partitions (e.g. both partitioned by hash on the join keys or by range with matching boundaries), switching to CollectLeft introduces an unnecessary CoalescePartitions and broadcasts data across partitions or cluster nodes.

Preserving PartitionMode::Partitioned when the inputs already satisfy the partitioned distribution requirements allows the join to execute locally within each partition with zero shuffles or network transfers.

What changes are included in this PR?

  • Added HashJoinExec::partitioned_input_distribution_requirements and HashJoinExec::inputs_satisfy_partitioned_requirements in datafusion/physical-plan/src/joins/hash_join/exec.rs.
  • In JoinSelection::try_collect_left (datafusion/physical-optimizer/src/join_selection.rs), skip converting to CollectLeft if inputs already satisfy partitioned distribution requirements across >1 partition.

What is the testing strategy for this PR?

New unit tests.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Sep 15, 2026
@stuhood
stuhood force-pushed the stuhood.do-not-broadcast-when-copartitioned branch from 1979696 to 4735398 Compare September 15, 2026 03:39
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Sep 15, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.13924% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (6bbd3f4) to head (4735398).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
...tafusion/physical-plan/src/joins/hash_join/exec.rs 92.10% 0 Missing and 6 partials ⚠️
...atafusion/physical-optimizer/src/join_selection.rs 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25304      +/-   ##
==========================================
- Coverage   81.91%   81.91%   -0.01%     
==========================================
  Files        1134     1134              
  Lines      425708   425722      +14     
  Branches   425708   425722      +14     
==========================================
+ Hits       348726   348729       +3     
  Misses      56305    56305              
- Partials    20677    20688      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@stuhood
stuhood marked this pull request as ready for review September 15, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JoinSelection should check if inputs already satisfy co-partitioning requirements before choosing CollectLeft (broadcast)

2 participants