Related:
Is your feature request related to a problem or challenge?
UnionExec does not preserve partitioning metadata. It concatenates child partitions and reports UnknownPartitioning(total_input_partitions), which is right because equal range keys from different children can land in different output partitions.
InterleaveExec is the partition-aware variant. This operator takes output partition i and combines partition i from each child. It preserves partitioning only when children have identical hash partitioning. This shoudl be extended to range partitioning
Example
InterleaveExec
DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3)
DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20)], 3)
Partition i of InterleaveExec combines partition i from each child, so compatible range boundaries can remain valid.
Describe the solution you'd like
Allow InterleaveExec to preserve Partitioning::Range when all children are compatible by partition index.
Acceptance criteria
- Keep
UnionExec conservative: it should continue to report unknown partitioning for concatenated child partitions.
- Remove the current SLT test that expects it to preserve range (I was mistaken originally on its behavior)
- Allow
InterleaveExec for compatible range children.
- Require co-partitioned children.
- Fall back to
UnknownPartitioning for mixed or incompatible inputs.
- Add tests for compatible range children, incompatible boundaries, and mixed partitioning.
Additional context
Related:
Is your feature request related to a problem or challenge?
UnionExecdoes not preserve partitioning metadata. It concatenates child partitions and reportsUnknownPartitioning(total_input_partitions), which is right because equal range keys from different children can land in different output partitions.InterleaveExecis the partition-aware variant. This operator takes output partitioniand combines partitionifrom each child. It preserves partitioning only when children have identical hash partitioning. This shoudl be extended to range partitioningExample
Partition
iofInterleaveExeccombines partitionifrom each child, so compatible range boundaries can remain valid.Describe the solution you'd like
Allow
InterleaveExecto preservePartitioning::Rangewhen all children are compatible by partition index.Acceptance criteria
UnionExecconservative: it should continue to report unknown partitioning for concatenated child partitions.InterleaveExecfor compatible range children.UnknownPartitioningfor mixed or incompatible inputs.Additional context