Skip to content

feat: allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements#23416

Draft
mithuncy wants to merge 2 commits into
apache:mainfrom
mithuncy:issue-23289-window-range-on-23184
Draft

feat: allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements#23416
mithuncy wants to merge 2 commits into
apache:mainfrom
mithuncy:issue-23289-window-range-on-23184

Conversation

@mithuncy

@mithuncy mithuncy commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Stacked on #23184 (first commit); only the last commit is this PR. Rebase and mark ready once #23184 merges. Overlaps with #23355, which also opts the window execs into range satisfaction as part of its PartitionedTopK work — happy to rebase whichever lands second.

What changes are included in this PR?

In one commit on top of #23184:

  • Opt WindowAggExec and BoundedWindowAggExec with partition keys into range satisfaction via InputDistributionRequirements::allow_range_satisfaction_for_key_partitioning, mirroring AggregateExec. Compatible range-partitioned inputs then satisfy the window key requirement without a hash repartition; subset satisfaction and the hash fallback for incompatible keys come from the existing satisfaction machinery. Windows without PARTITION BY keep requiring a single partition.

Are these changes tested?

Yes: new slt tests in range_partitioning.slt (exact and subset reuse, rehash on incompatible keys, subset_repartition_threshold / preserve_file_partitions / target_partitions behavior, WindowAggExec via an unbounded frame, and no-PARTITION BY), plus plan-shape tests in enforce_distribution.rs and acceptance/rejection tests in sanity_checker.rs.

Are there any user-facing changes?

No: only plan changes.

gene-bordegaray and others added 2 commits July 8, 2026 07:39
…rtitioned requirements

Windows got a hash repartition even when the input was already range
partitioned on the PARTITION BY keys. Opt WindowAggExec and
BoundedWindowAggExec into range satisfaction for KeyPartitioned, the
same way AggregateExec does. Incompatible keys still fall back to hash
repartitioning, and windows without PARTITION BY keep requiring a
single partition.

Covered by enforce_distribution and sanity_checker tests plus a window
section in range_partitioning.slt.

Closes apache#23289
@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion v54.0.0 (current)
       Built [ 101.901s] (current)
     Parsing datafusion v54.0.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion v54.0.0 (baseline)
       Built [ 101.040s] (baseline)
     Parsing datafusion v54.0.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.625s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 205.594s] datafusion
    Building datafusion-datasource v54.0.0 (current)
       Built [  37.461s] (current)
     Parsing datafusion-datasource v54.0.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-datasource v54.0.0 (baseline)
       Built [  37.587s] (baseline)
     Parsing datafusion-datasource v54.0.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion-datasource v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.245s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  76.676s] datafusion-datasource
    Building datafusion-physical-expr v54.0.0 (current)
       Built [  28.629s] (current)
     Parsing datafusion-physical-expr v54.0.0 (current)
      Parsed [   0.045s] (current)
    Building datafusion-physical-expr v54.0.0 (baseline)
       Built [  28.958s] (baseline)
     Parsing datafusion-physical-expr v54.0.0 (baseline)
      Parsed [   0.047s] (baseline)
    Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.340s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron

Failed in:
  RangePartitioning::compatible_with, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/40bec6fa31fb8209d998577a194058efbe5f43b7/datafusion/physical-expr/src/partitioning.rs:253
  Partitioning::compatible_with, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/40bec6fa31fb8209d998577a194058efbe5f43b7/datafusion/physical-expr/src/partitioning.rs:416

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  58.786s] datafusion-physical-expr
    Building datafusion-physical-optimizer v54.0.0 (current)
       Built [  39.043s] (current)
     Parsing datafusion-physical-optimizer v54.0.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-physical-optimizer v54.0.0 (baseline)
       Built [  38.762s] (baseline)
     Parsing datafusion-physical-optimizer v54.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-physical-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.114s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  78.920s] datafusion-physical-optimizer
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  36.486s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.140s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  36.906s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.136s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.642s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---

Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_marked_deprecated.ron

Failed in:
  method required_input_distribution in trait datafusion_physical_plan::execution_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98
  method required_input_distribution in trait datafusion_physical_plan::ExecutionPlan in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/execution_plan.rs:98

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  75.401s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 176.834s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 176.343s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.088s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 355.910s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jul 9, 2026
@gene-bordegaray

Copy link
Copy Markdown
Contributor

This is in my queue, just waiting for #23184 to land 🙇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change core Core DataFusion crate datasource Changes to the datasource crate optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow Partitioning::Range to satisfy window Distribution::KeyPartitioned requirements

2 participants