Is your feature request related to a problem or challenge?
Follow-up from #23189
File scans for Hive/value-partitioned data currently represent their output partitioning as Partitioning::Hash on the partition columns. This lets the optimizer avoid repartitioning for key-partitioned operators, but the data was
not actually assigned to partitions by DataFusion's hash partitioner.
This makes Partitioning::Hash carry multiple meanings:
- DataFusion hash partitioning with a particular hash implementation / seed
- another hash implementation / seed
- Hive/value-based file grouping by partition column values
- potentially other source-defined key partitioning methods
These partitioning methods can all provide key colocation, but they are not necessarily co-partitioned by partition index. Treating them all as Partitioning::Hash can make later optimizer decisions and dynamic filtering
decisions brittle or incorrect if they rely on partition compatibility.
Describe the solution you'd like
Discuss and design a more precise representation for value-based physical partitioning. Possible directions from the PR discussion include:
- add a new
Partitioning::Value / ValuePartitioned variant
- represent Hive/value partitioning as
Partitioning::Range when values and ordering are available
- make the partitioning method explicit, e.g. hash algorithm/seed vs column value grouping
- explore whether
Partitioning should become extensible for source-defined partitioning methods
The design should distinguish:
- satisfying a
Distribution::KeyPartitioned requirement, where key colocation may be enough
- proving co-partitioning / partition-index compatibility between two inputs, where the exact partitioning method and value-to-partition mapping matter
Describe alternatives you've considered
No response
Additional context
Related issues and PRs:
Relevant review thread:
CC @gene-bordegaray @alamb @gabotechs @jayshrivastava
Is your feature request related to a problem or challenge?
Follow-up from #23189
File scans for Hive/value-partitioned data currently represent their output partitioning as
Partitioning::Hashon the partition columns. This lets the optimizer avoid repartitioning for key-partitioned operators, but the data wasnot actually assigned to partitions by DataFusion's hash partitioner.
This makes
Partitioning::Hashcarry multiple meanings:These partitioning methods can all provide key colocation, but they are not necessarily co-partitioned by partition index. Treating them all as
Partitioning::Hashcan make later optimizer decisions and dynamic filteringdecisions brittle or incorrect if they rely on partition compatibility.
Describe the solution you'd like
Discuss and design a more precise representation for value-based physical partitioning. Possible directions from the PR discussion include:
Partitioning::Value/ValuePartitionedvariantPartitioning::Rangewhen values and ordering are availablePartitioningshould become extensible for source-defined partitioning methodsThe design should distinguish:
Distribution::KeyPartitionedrequirement, where key colocation may be enoughDescribe alternatives you've considered
No response
Additional context
Related issues and PRs:
Partitioning::Rangeto satisfyDistribution::KeyPartitionedgenerally #23266Relevant review thread:
CC @gene-bordegaray @alamb @gabotechs @jayshrivastava