Skip to content

Audit multi-child Distribution::KeyPartitioned requirements before general Range satisfaction #23451

Description

@gene-bordegaray

Related:

Is your feature request related to a problem or challenge?

General Partitioning::Range satisfaction for Distribution::KeyPartitioned is tracked by #23266. Multi-child operators are harder than single-input operators because independently key-partitioned inputs are not always co-partitioned by partition index see #23184 for more contect on co-partitioning.

Some multi-child operators still declare per-child KeyPartitioned requirements directly. Before range satisfaction becomes general, these operators should either use a co-partitioning requirement to guard them to not be satisfied by independent children requirements alone.

Example

SortMergeJoinExec: join_type=Inner, on=[(range_key@0, range_key@0)]
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4)
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4)

Each input may independently satisfy KeyPartitioned(range_key), but the split points are not compatible. A partition-index aware operator needs co-partitioning proof before it can skip repartitioning.

Describe the solution you'd like

Audit multi-child physical operators that declare KeyPartitioned input requirements and convert partition-index aware operators to InputDistributionRequirements::co_partitioned(...).

Operators to audit (please do your own pass as well):

  • SortMergeJoinExec
  • SymmetricHashJoinExec

Acceptance criteria

  • Partition-index aware multi-child operators use explicit co-partitioning requirements.
  • Compatible range/range inputs avoid repartitioning only when cross-child boundaries, partition counts, sort options, and key expressions are compatible.
  • Incompatible range/range inputs require repartitioning path.
  • Tests cover independently range-partitioned inputs with incompatible cross-child distributions.

Additional context

This should happen before or as part of #23266 so general range satisfaction does not accidentally treat per-child key colocation as cross-child co-partitioning.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions