Skip to content

Allow InterleaveExec to preserve Partitioning::Range #23455

Description

@gene-bordegaray

Related:

Is your feature request related to a problem or challenge?

UnionExec does not preserve partitioning metadata. It concatenates child partitions and reports UnknownPartitioning(total_input_partitions), which is right because equal range keys from different children can land in different output partitions.

InterleaveExec is the partition-aware variant. This operator takes output partition i and combines partition i from each child. It preserves partitioning only when children have identical hash partitioning. This shoudl be extended to range partitioning

Example

InterleaveExec
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3)
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3)

Partition i of InterleaveExec combines partition i from each child, so compatible range boundaries can remain valid.

Describe the solution you'd like

Allow InterleaveExec to preserve Partitioning::Range when all children are compatible by partition index.

Acceptance criteria

  • Keep UnionExec conservative: it should continue to report unknown partitioning for concatenated child partitions.
  • Remove the current SLT test that expects it to preserve range (I was mistaken originally on its behavior)
  • Allow InterleaveExec for compatible range children.
  • Require co-partitioned children.
  • Fall back to UnknownPartitioning for mixed or incompatible inputs.
  • Add tests for compatible range children, incompatible boundaries, and mixed partitioning.

Additional context

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions