From 104d549009db7796a2185ea585c431d79a9072a7 Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Thu, 3 Sep 2026 15:37:24 +0200 Subject: [PATCH 1/6] [SPARK-59620][SQL] Late-materialization storage-filter pushdown in the vectorized Parquet reader ### What changes were proposed in this pull request? This adds a late-materialization read path to the vectorized Parquet reader, so the optimizer can push a runtime filter into the scan and have it prune value-column IO instead of running as a post-scan `FilterExec`. The filter it pushes today is `BloomFilterMightContain`, the runtime bloom `InjectRuntimeFilter` builds for a join. The reader reads the filter's key columns first, evaluates the predicate per row, and then reads the remaining columns restricted to the surviving row ranges. Each output batch is spliced together from the key vectors it kept and the value vectors it read for those rows. **Planning.** - New SQL conf `spark.sql.parquet.storageFilterPushdown.enabled`, default `false`, session-bound. It is a planning-time decision only. With it off, no storage filter is attached to a scan in the first place and the bloom stays where it is today. - `FileSourceStrategy.extractStorageFilters` lifts eligible top-level `BloomFilterMightContain` conjuncts out of `afterScanFilters` into a new `storageFilters` slot on `FileSourceScanExec`. Eligibility is checked in full at planning time, because extraction removes the conjunct from the post-scan `Filter` and nothing else would apply it afterwards. A conjunct qualifies when the file format is exactly `ParquetFileFormat`, the vectorized reader is feasible for `partitionSchema ++ outputDataSchema`, the conjunct is deterministic, and every reference on the bloom's value side is a projected data column of a type the reader can copy. - `ParquetStorageFilter.isSupportedKeyType` is the single authority on key-column types. Both `extractStorageFilters` and `ParquetStorageFilter.create` consult it, and it lists exactly the types `VectorizedParquetRecordReader.copierFor` handles. It is deliberately narrower than `AtomicType`, since `VariantType`, `GeometryType` and `GeographyType` are atomic but have no primitive Parquet leaf for phase 1 to read into. - `FileSourceScanLike` gains `storageFilters: Seq[Expression]` and five SQL metrics, described below. The `StorageFilters` entry in the scan description is only emitted when the scan has storage filters, so explain output is unchanged for everyone else. - `FileSourceScanExec.preparedStorageFilters` materializes scalar subqueries and binds attributes to `BoundReference`s indexing `requiredSchema`. The conf is deliberately not rechecked at execution time. Once extraction has dropped a bloom from the post-scan `Filter`, the runtime has to honour that decision or fail, and it does the second. **`FileFormat` API.** - A new `buildReaderWithStorageFilters` overload takes `storageFilters: Seq[Expression]` and the metric map. Its default body requires `storageFilters` to be empty and otherwise delegates to `buildReaderWithPartitionValues`, so a format that does not implement the feature rejects a filter it cannot honour rather than dropping it. - `FileSourceScanExec.inputRDD` only routes through the new entry point when there is something to push, so a `ParquetFileFormat` subclass that customizes reading by overriding `buildReaderWithPartitionValues` keeps working unchanged. **Parquet reader.** - A new `ParquetStorageFilter` value object holds the bound expressions, the key-column indices into the requested schema, and the optional metrics. `rewriteForMissingKeys` and `evalAllMissing` handle schema evolution, where a key column is in the requested schema but absent from the physical file. The rewritten predicate is evaluated against the value the reader will actually materialize for that column, which is the column's existence `DEFAULT` when it has one and null otherwise. Evaluating against null instead would filter on a value the scan never returns, and `XxHash64` is `nullable = false` so a null input hashes to the seed rather than producing null. - `SpecificParquetRecordReaderBase` exposes the underlying `ParquetFileReader`, the input file and the footer, so the late-materialization driver can switch the requested schema per phase and call `readFilteredRowGroup(blockIdx, rowRanges)`. - `VectorizedParquetRecordReader` gains a three-phase per-row-group loop driven by that one reader. Phase 0 computes `pushedFilterRanges` from the pushed data filter through the column index, which is metadata only. Phase 1 switches to the key-only schema, reads the key columns under those ranges, evaluates the storage filter per row, and accumulates the survivors into per-key-column queues of capacity-sized `WritableColumnVector`s. Phase 2 switches to the non-key schema and reads those columns under the surviving ranges. Emit splices the dequeued key vectors with the freshly read non-key vectors in the projection's own order. - A projection that is all key columns skips phase 2 entirely and reconstructs every batch from the key queues, which is the shape with the largest saving. - Phase 0 checks `parquet.filter.columnindex.enabled` itself, because `ParquetFileReader.getRowRanges` only asks whether a filter is pushed. That conf is the documented escape hatch for a file whose column index is wrong, and trusting a wrong column index here would drop rows for good. - `requireOffsetIndexesForPhase2` fails at reader init if any projected column of any row group has no offset index. Phase 2 reads a strict subset of a row group's rows, which parquet can only do through the offset index, and neither widening the read nor skipping the filter is correct. The check reads only footer fields, and it covers every projected column rather than the non-key ones alone, because parquet builds one column index store per row group and returns an empty one as soon as any path in it lacks an offset index. - Per-key-column value copying uses a `ValueCopier` chosen once at init, so the survivor loop has no per-value type dispatch. - `initBatch` threads a `skipDataSlots` set through `allocateColumns`, so the persistent output vectors for key slots are never allocated. Under splicing those slots come from the queues. - Preconditions that planning already guarantees throw instead of falling back, for the same reason the conf is not rechecked. That covers a key ordinal out of range, a non-primitive key column, missing key pages, and a vectorized-reader conf flipped between planning and execution. The two remaining silent fallbacks cannot change the result, namely a reader with no underlying `ParquetFileReader`, which only test mocks produce, and a file where every key column is missing, which is answered by evaluating the rewritten constant predicate. **Metrics.** Five, all created only when the scan has storage filters, and all scoped to what the storage filter added on top of a no-storage-filter read of the same projection. Each counter names its own quantity, so the three verbs are deliberate. A row group is skipped, meaning its data columns were never read while phase 1 did read its key columns. A row is excluded, meaning it never reached the output. A byte is avoided, meaning it was never transferred. - `storageFilterRowGroupsSkipped`, "row groups skipped by storage filter". - `storageFilterRowsExcludedByRowGroup`, "rows excluded by storage filter (whole row group)". - `storageFilterRowsExcludedWithinRowGroup`, "rows excluded by storage filter (within row group)". The suffix says where the row was excluded rather than by which mechanism, because a row sharing a page with a survivor is read and dropped during decode. - `storageFilterBytesAvoidedByRowGroup`, "bytes avoided by storage filter (whole row group)". - `storageFilterBytesAvoidedByPageFiltering`, "bytes avoided by storage filter (page filtering)". The byte counters must not cost IO to report, so `compressedBytesForRowRanges` answers from the footer's `ColumnChunkMetaData.getTotalSize()` whenever the row range covers the whole block, and walks the offset index only for a strict subset. That is complete rather than a mitigation. A range narrower than the block can only come from column-index filtering, which builds and memoizes the store as a side effect, and phase 2's own read builds it before the walk in the other case. The walk counts the dictionary page too, since parquet reads it whenever it reads any data page of a chunk. ### Why are the changes needed? A runtime bloom filter from join runtime filtering is applied as a post-scan `FilterExec` today. The scan still reads every value page of every row group, even where the bloom drops almost every row immediately. On a selective join over a wide table that read is the dominant cost. Late materialization turns that around. The scan reads the bloom's key column first, decides which rows survive, and never reads the value pages no surviving row touches. A row group where nothing survives costs one key-column read and no value IO at all. ### Does this PR introduce _any_ user-facing change? No change by default, since the conf is off and nothing is attached to a scan in that case. With the conf on: - Queries return the same rows. The late-materialization path is exact, and a filter that fails any eligibility check keeps its existing post-scan `FilterExec`. - `FileSourceScanExec` reports the five metrics above in the SQL UI, and its description gains a `StorageFilters` entry. - A scan reading a Parquet file written without offset indexes fails with an error naming the conf to turn off. Files written by parquet-mr 1.11 and later always have them. - A consumer that illegally retains a `ColumnarBatch` across `next()` sees a sharper edge than before. On the plain path the previous batch's key vectors are reused, and under splicing they are freed at the next `nextBatch()`, so with off-heap vectors the retained reference points at released memory rather than stale values. The contract already forbids retaining a batch. - Phase 1 buffers a whole row group's surviving key values before it produces that row group's first batch, so a task holds up to one extra copy of the key columns for one row group. The conf's documentation says so. ### How was this patch tested? New `ParquetStorageFilterSuite`, 89 tests. Four blind review rounds ran over the change, each by an agent with no knowledge of the earlier ones, and every finding is fixed in this commit. - Reader-level tests over hand-built filters: whole row group rejected, nothing rejected, mixed, multi-batch emit, key-only projection, a survivor count that is an exact multiple of the batch capacity, and a projection whose key column is not in the leading slot. - One case per `ValueCopier` branch across 17 types and both encodings, each comparing the splicing path against the plain reader over the same file, so values are checked and not only row counts. `isSupportedKeyType` is asserted to cover exactly the copier's types. - Nullable keys, two key columns, partition columns alongside spliced keys, off-heap vectors, the row-at-a-time path, `_metadata.row_index`, and a complex non-key column. - Schema evolution: `rewriteForMissingKeys` and `evalAllMissing` as units, plus end-to-end reads where a key column is missing from the older file with a `DEFAULT`, with a `DEFAULT` that fails the filter, and with no `DEFAULT`. - Page-level ranges, which need a multi-page row group: phase 1 stays aligned with the row indexes, and the byte metrics stay non-negative. - Metric arithmetic: emitted plus excluded accounts for every row of the file, and an all-key projection reports zero avoided bytes. - Planner gates: the bloom stays in the post-scan `Filter` when the vectorized reader is unavailable, when the conf is off, and when the conjunct is non-deterministic. A scan whose vectorized reader is disabled after planning fails loudly. Extraction preserves results with AQE on and off. Canonicalization keeps a storage-filter scan distinct from a plain one, so exchange and subquery reuse cannot cross them. - Whole-stage codegen off, where the planner's gate is weaker than the runtime's, so the bloom is extracted and the spliced batch is served one row at a time. - Column-index filtering off, the branch that decides where phase 0's ranges come from. The row accounting tells the two arms apart, since with the column index off every row of the block reaches phase 1. - A limit under whole-stage codegen, which is the only shape that closes the spliced batch from outside while the reader is still open. The test asserts the generated source contains that close, so it cannot pass for the wrong reason. - The byte metrics cost no extra IO, measured rather than argued. The same read runs twice over a filesystem that counts every byte handed back, once with all five metrics wired and once with none, and the counts match. Both range shapes are covered, the offset-index one and the footer one. Knowingly untested, both for the same reason. The offset-index check's throw needs a file with no offset index, which parquet-mr cannot write. The `ParquetFileFormat`-subclass bypass needs a third-party subclass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code with Claude Opus 4.7 and Claude Opus 5 Co-authored-by: Matt Butrovich --- .../apache/spark/sql/internal/SQLConf.scala | 18 + .../SpecificParquetRecordReaderBase.java | 38 +- .../VectorizedParquetRecordReader.java | 901 +++++++- .../sql/execution/DataSourceScanExec.scala | 117 +- .../execution/datasources/FileFormat.scala | 42 + .../datasources/FileSourceStrategy.scala | 77 +- .../parquet/ParquetFileFormat.scala | 58 +- .../parquet/ParquetStorageFilter.scala | 220 ++ .../apache/spark/sql/DataFrameJoinSuite.scala | 3 +- .../org/apache/spark/sql/SubquerySuite.scala | 2 +- .../parquet/ParquetStorageFilterSuite.scala | 1965 +++++++++++++++++ 11 files changed, 3411 insertions(+), 30 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 110b030c09ba8..3c2f18eeb5d9c 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1885,6 +1885,21 @@ object SQLConf { .booleanConf .createWithDefault(true) + val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED = + buildConf("spark.sql.parquet.storageFilterPushdown.enabled") + .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " + + "(e.g. bloom filters from join runtime filtering) at the scan level using late " + + "materialization: read key columns first, evaluate the filter per row, then read data " + + "columns restricted to surviving rows. This is a planning-time decision only: when " + + "false, no storage filter is attached to a scan in the first place and the filter is " + + "applied as an ordinary post-scan filter instead. Note that the surviving key values of " + + "a whole row group are buffered before the first batch of that row group is produced, " + + "so a task holds up to one extra copy of the key columns for one row group.") + .version("5.0.0") + .withBindingPolicy(ConfigBindingPolicy.SESSION) + .booleanConf + .createWithDefault(false) + val PARQUET_FILTER_PUSHDOWN_DATE_ENABLED = buildConf("spark.sql.parquet.filterPushdown.date") .doc("If true, enables Parquet filter push-down optimization for Date. " + s"This configuration only has an effect when '${PARQUET_FILTER_PUSHDOWN_ENABLED.key}' is " + @@ -9106,6 +9121,9 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def parquetFilterPushDown: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_ENABLED) + def parquetStorageFilterPushdownEnabled: Boolean = + getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED) + def parquetFilterPushDownDate: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_DATE_ENABLED) def parquetFilterPushDownTimestamp: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_TIMESTAMP_ENABLED) diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java index eb0063688e701..11fe908eaa10b 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java @@ -87,6 +87,13 @@ public abstract class SpecificParquetRecordReaderBase extends RecordReader columns) throws IOException .builder(configuration, file) .withRange(0, length) .build(); - ParquetFileReader fileReader = ParquetFileReader.open( - HadoopInputFile.fromPath(file, configuration), options); + this.inputFile = HadoopInputFile.fromPath(file, configuration); + ParquetFileReader fileReader = ParquetFileReader.open(this.inputFile, options); + this.fileFooter = fileReader.getFooter(); this.reader = new ParquetRowGroupReaderImpl(fileReader); - this.fileSchema = fileReader.getFooter().getFileMetaData().getSchema(); + this.fileSchema = fileFooter.getFileMetaData().getSchema(); if (columns == null) { this.requestedSchema = fileSchema; @@ -278,6 +289,16 @@ interface ParquetRowGroupReader extends Closeable { * Reads the next row group from this reader. Returns null if there is no more row group. */ PageReadStore readNextRowGroup() throws IOException; + + /** + * Returns the underlying {@link ParquetFileReader}, or null if this reader does not wrap one + * (e.g. test implementations). Callers can use this to access lower-level APIs such as + * {@code setRequestedSchema}, {@code readRowGroup(int)} and + * {@code readFilteredRowGroup(int, RowRanges)} which are needed for late materialization. + */ + default ParquetFileReader getUnderlyingReader() { + return null; + } } private static class ParquetRowGroupReaderImpl implements ParquetRowGroupReader { @@ -292,6 +313,11 @@ public PageReadStore readNextRowGroup() throws IOException { return reader.readNextFilteredRowGroup(); } + @Override + public ParquetFileReader getUnderlyingReader() { + return reader; + } + @Override public void close() throws IOException { if (reader != null) { diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index 5e782433f5576..2252dd93b5564 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -19,9 +19,13 @@ import java.io.IOException; import java.time.ZoneId; +import java.util.ArrayList; import java.util.Arrays; +import java.util.HashMap; import java.util.HashSet; import java.util.List; +import java.util.Map; +import java.util.PrimitiveIterator; import java.util.Set; import scala.Option; @@ -32,17 +36,28 @@ import org.apache.hadoop.mapreduce.TaskAttemptContext; import org.apache.parquet.column.ColumnDescriptor; import org.apache.parquet.column.page.PageReadStore; +import org.apache.parquet.filter2.columnindex.RowRanges; +import org.apache.parquet.hadoop.ParquetFileReader; +import org.apache.parquet.hadoop.ParquetInputFormat; +import org.apache.parquet.hadoop.metadata.BlockMetaData; +import org.apache.parquet.hadoop.metadata.ColumnChunkMetaData; +import org.apache.parquet.hadoop.metadata.ColumnPath; import org.apache.parquet.hadoop.metadata.ParquetMetadata; import org.apache.parquet.hadoop.util.HadoopInputFile; +import org.apache.parquet.internal.column.columnindex.OffsetIndex; +import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore; +import org.apache.parquet.internal.filter2.columnindex.ColumnIndexStore.MissingOffsetIndexException; import org.apache.parquet.io.SeekableInputStream; import org.apache.parquet.schema.GroupType; import org.apache.parquet.schema.MessageType; import org.apache.parquet.schema.Type; +import org.apache.parquet.schema.Types; import org.apache.spark.SparkUnsupportedOperationException; import org.apache.spark.memory.MemoryMode; import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns; import org.apache.spark.sql.catalyst.InternalRow; +import org.apache.spark.sql.execution.metric.SQLMetric; import org.apache.spark.sql.execution.vectorized.ColumnVectorUtils; import org.apache.spark.sql.execution.vectorized.ConstantColumnVector; import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector; @@ -149,6 +164,75 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa */ private final MemoryMode MEMORY_MODE; + /** + * Optional storage filter for late materialization: read key-column pages first, evaluate the + * filter per row to build {@link RowRanges}, then read data-column pages restricted to surviving + * row ranges. Null means the normal (eager) read path is used. + */ + private ParquetStorageFilter storageFilter; + + /** + * Set to true once all row groups have been processed (used with late materialization). + * */ + private boolean hitEndOfData = false; + + /** + * Late-materialization state, populated by {@link #initializeLateMaterialization()}. + * + *

One {@link ParquetFileReader} ({@link #lateMatReader}, the base class's reader exposed via + * {@link ParquetRowGroupReader#getUnderlyingReader()}) drives all three phases. Its requested + * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: full schema for + * phase 0 ({@code getRowRanges}), key-only for phase 1, non-key only for phase 2 (or skipped + * entirely when the projected schema is all keys, indicated by {@link #nonKeyRequestedSchema} + * being null). + */ + private ParquetFileReader lateMatReader; + private MessageType keyOnlyRequestedSchema; + private MessageType nonKeyRequestedSchema; + private int nextBlockIndex; + private int totalBlockCount; + private ColumnDescriptor[] keyDescriptors; + private boolean[] keyRequired; + private WritableColumnVector[] keyScratchVectors; + private ColumnarBatch keyScratchBatch; + /** + * Mirrors parquet's {@code ParquetReadOptions.useColumnIndexFilter()}. Phase 0 consults it + * because {@link ParquetFileReader#getRowRanges(int)} does not: it checks only whether a filter + * is pushed, so it would keep narrowing by column index after a user disabled that filtering. + */ + private boolean useColumnIndexFilter = true; + + /** + * Splicing state. {@link #isKeyTopLevel} marks top-level requested-schema slots that are sourced + * from the per-key-column queues instead of phase-2 reads. + * {@link #keyVectorQueues} holds one queue per (present) key column of capacity-sized survivor + * vectors filled during phase 1; {@link #currentKeyAccumulators} are the currently-filling + * vectors not yet pushed to the queue. + * {@link #pendingCloseKeyVectors} stashes the previous emit's dequeued vectors so they can be + * closed at the start of the next emit, releasing survivor memory incrementally as we emit. + * {@link #persistentBatchColumns} captures the original {@link #initBatch} vector array so it can + * be closed separately from the per-emit splicing batch (which shares those vectors and would + * otherwise double-close them). + * + *

Phase 2 is skipped entirely when the projected schema is all key columns + * ({@link #nonKeyRequestedSchema} is null); emit reconstructs each batch purely from the key + * queues. + * + *

Memory: phase 1 evaluates the whole row group before the first batch of that row group is + * emitted, so the queues hold every surviving key value for one row group at once -- up to one + * extra copy of the key columns per row group, versus one capacity-sized vector on the plain read + * path. {@link #pendingCloseKeyVectors} then releases them batch by batch as emit progresses. + */ + private boolean[] isKeyTopLevel; + private java.util.ArrayDeque[] keyVectorQueues; + private WritableColumnVector[] currentKeyAccumulators; + /** Row count of {@link #currentKeyAccumulators}; all key columns advance in lockstep. */ + private int currentKeyAccumulatorRowCount; + /** Per-key-column copier picked once at init time; called per surviving row in the hot loop. */ + private ValueCopier[] keyCopiers; + private WritableColumnVector[] pendingCloseKeyVectors; + private ColumnVector[] persistentBatchColumns; + public VectorizedParquetRecordReader( ZoneId convertTz, String datetimeRebaseMode, @@ -224,11 +308,44 @@ public void initialize( @Override public void close() throws IOException { - if (columnarBatch != null) { - columnarBatch.close(); - columnarBatch = null; + // Each release below is independent, and super.close() owns the file handle and input + // stream, so they are chained through finally blocks: one failing vector close must not leak + // the rest. + try { + if (isKeyTopLevel != null) { + // Splicing: the per-emit columnarBatch is a transient view whose slots alias + // persistentBatchColumns (non-key + partition) and pendingCloseKeyVectors (dequeued key + // slots). We never call columnarBatch.close() because that would re-close those shared + // vectors after the direct closes below, freeing the same buffer twice. + try { + if (persistentBatchColumns != null) { + for (ColumnVector v : persistentBatchColumns) { + if (v != null) v.close(); + } + persistentBatchColumns = null; + } + columnarBatch = null; + } finally { + closeSplicingState(); + } + } else if (columnarBatch != null) { + columnarBatch.close(); + columnarBatch = null; + persistentBatchColumns = null; + } + } finally { + try { + if (keyScratchBatch != null) { + keyScratchBatch.close(); + keyScratchBatch = null; + keyScratchVectors = null; + } + } finally { + // lateMatReader aliases the base-class reader; super.close() owns it. + lateMatReader = null; + super.close(); + } } - super.close(); } @Override @@ -252,6 +369,9 @@ public Object getCurrentValue() { @Override public float getProgress() { + // Under a storage filter, rowsReturned counts survivors while totalRowCount is the pre-filter + // count, so the ratio would stall below 1. hitEndOfData is the real terminator there. + if (hitEndOfData) return 1.0f; return (float) rowsReturned / totalRowCount; } @@ -283,13 +403,21 @@ private void initBatch( constantColumnLength = partitionColumns.fields().length; } + Set keySlotsToSkip = null; + if (isKeyTopLevel != null) { + int[] keyIndices = storageFilter.keyColumnIndices(); + keySlotsToSkip = new HashSet<>(keyIndices.length); + for (int idx : keyIndices) keySlotsToSkip.add(idx); + } ColumnVector[] vectors = allocateColumns( - capacity, batchSchema, memMode == MemoryMode.OFF_HEAP, constantColumnLength); + capacity, batchSchema, memMode == MemoryMode.OFF_HEAP, constantColumnLength, keySlotsToSkip); columnarBatch = new ColumnarBatch(vectors); + persistentBatchColumns = vectors; columnVectors = new ParquetColumnVector[sparkSchema.fields().length]; for (int i = 0; i < columnVectors.length; i++) { + if (vectors[i] == null) continue; // splicing key slot; ParquetColumnVector unused Object defaultValue = null; if (sparkRequestedSchema != null) { defaultValue = ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema)[i]; @@ -391,12 +519,15 @@ public void enableReturningBatches() { * Advances to the next batch of rows. Returns false if there are no more. */ public boolean nextBatch() throws IOException { + if (isKeyTopLevel != null) return nextBatchSplicing(); for (ParquetColumnVector vector : columnVectors) { vector.reset(); } columnarBatch.setNumRows(0); + if (hitEndOfData) return false; if (rowsReturned >= totalRowCount) return false; checkEndOfRowGroup(); + if (hitEndOfData) return false; int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); for (ParquetColumnVector cv : columnVectors) { @@ -421,11 +552,313 @@ public boolean nextBatch() throws IOException { return true; } + /** + * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory + * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()}, + * dequeues one survivor key vector per key column, drives non-key column readers for {@code num} + * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key + * (persistent value-vector) slots in the original projection order. + * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and + * {@link #closeSplicingState()}. + */ + private boolean nextBatchSplicing() throws IOException { + if (pendingCloseKeyVectors != null) { + for (WritableColumnVector v : pendingCloseKeyVectors) { + if (v != null) v.close(); + } + pendingCloseKeyVectors = null; + } + for (int i = 0; i < columnVectors.length; i++) { + if (isKeyTopLevel[i]) continue; + columnVectors[i].reset(); + } + // Match the eager path and zero the outgoing batch before the terminal checks below. Without + // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots + // were just closed above -- with off-heap vectors those buffers are already freed, so a + // consumer that read the batch after nextBatch() returned false would see freed memory. + if (columnarBatch != null) columnarBatch.setNumRows(0); + if (hitEndOfData) return false; + if (rowsReturned >= totalRowCount) return false; + checkEndOfRowGroup(); + if (hitEndOfData) return false; + + int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); + + WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length]; + for (int i = 0; i < keyVectorQueues.length; i++) { + dequeued[i] = keyVectorQueues[i].removeFirst(); + } + + for (int i = 0; i < columnVectors.length; i++) { + if (isKeyTopLevel[i]) continue; + ParquetColumnVector cv = columnVectors[i]; + for (ParquetColumnVector leafCv : cv.getLeaves()) { + VectorizedColumnReader columnReader = leafCv.getColumnReader(); + if (columnReader != null) { + columnReader.readBatch(num, leafCv.getValueVector(), + leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector()); + } + } + cv.assemble(); + } + if (rowIndexGenerator != null) { + // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a + // synthetic metadata column never referenced by a storage filter, so its slot is a non-key + // slot with a persistent ParquetColumnVector. + rowIndexGenerator.populateRowIndex(columnVectors, num); + } + + ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length]; + // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in + // key-row-position order, so it pairs the k-th smallest key slot with key-row position k. + // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices` + // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their + // position in that list, so this loop cannot reconstruct the pairing on its own: if the list + // ever stops being sorted, key columns silently swap places in the output batch. + int keyIdx = 0; + for (int i = 0; i < persistentBatchColumns.length; i++) { + if (i < isKeyTopLevel.length && isKeyTopLevel[i]) { + cols[i] = dequeued[keyIdx++]; + } else { + cols[i] = persistentBatchColumns[i]; + } + } + columnarBatch = new ColumnarBatch(cols); + columnarBatch.setNumRows(num); + + rowsReturned += num; + numBatched = num; + batchIdx = 0; + pendingCloseKeyVectors = dequeued; + return true; + } + private void initializeInternal() throws IOException, UnsupportedOperationException { missingColumns = new HashSet<>(); for (ParquetColumn column : CollectionConverters.asJava(parquetColumn.children())) { checkColumn(column); } + if (storageFilter != null) { + initializeLateMaterialization(); + } + } + + /** + * Sets the storage filter for late materialization. Must be called before {@link #initialize}; + * {@link #initializeLateMaterialization()} (run from {@link #initialize}) inspects the per-file + * schema and decides whether splicing actually engages. + * + *

Splicing does NOT engage in one real case: all key columns are missing from this physical + * file under schema evolution. The predicate is rewritten with each missing key replaced by the + * constant the reader materializes for it (its existence DEFAULT, else null -- see + * {@code ParquetStorageFilter.rewriteForMissingKeys}) and evaluated as a constant: a true result + * keeps the file with no filtering, a false/null result skips it entirely. Either way the rows + * the scan returns are exactly the rows that satisfy the filter, so this is safe. + * + *

Every OTHER precondition is guaranteed by planning-time checks in + * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and + * violations throw rather than fall back: once extraction has moved a bloom filter onto the scan + * it is gone from the post-scan Filter, so quietly not applying it would produce wrong rows. + */ + public void setStorageFilter(ParquetStorageFilter storageFilter) { + this.storageFilter = storageFilter; + } + + private void initializeLateMaterialization() throws IOException { + lateMatReader = reader.getUnderlyingReader(); + if (lateMatReader == null) { + // Unreachable in production: the only ParquetRowGroupReader ParquetFileFormat builds is + // ParquetRowGroupReaderImpl, which returns its reader. Dropping the filter here would return + // rows it rejects, since extraction already removed it from the post-scan Filter. + throw new IllegalStateException( + "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but " + + reader.getClass().getName() + " does not expose one"); + } + if (configuration != null) { + useColumnIndexFilter = configuration.getBoolean( + ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true); + } + + // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the + // latter can happen under schema evolution: a column is in the requested schema but not in this + // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive + // leaves. + int[] keyIndices = storageFilter.keyColumnIndices(); + List presentKeyColumns = new ArrayList<>(keyIndices.length); + List missingKeyLocalPositions = new ArrayList<>(); + for (int i = 0; i < keyIndices.length; i++) { + int idx = keyIndices[i]; + if (idx < 0 || idx >= parquetColumn.children().size()) { + // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. Fail loudly + // rather than dropping the filter -- extractStorageFilters already removed it from the + // post-scan Filter, so silently ignoring it here would return wrong rows. + throw new IllegalStateException(String.format( + "Storage-filter key ordinal %d is out of range for a %d-column requested schema", + idx, parquetColumn.children().size())); + } + ParquetColumn column = parquetColumn.children().apply(idx); + if (!column.isPrimitive()) { + // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive + // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. Fail loudly for + // the same reason as above. + throw new IllegalStateException( + "Storage-filter key column is not a primitive Parquet column: " + column.path()); + } + if (missingColumns.contains(column)) { + missingKeyLocalPositions.add(i); + } else { + presentKeyColumns.add(column); + } + } + + // If any key column is missing from this file, rewrite the predicate to substitute the constant + // the reader will actually materialize for that column. That is the column's existence DEFAULT + // when it has one (ParquetColumnVector writes it into the output vector and marks the vector + // constant), otherwise null. Substituting null for a column that reads back as its default + // would filter on a value the scan never returns. The predicate must be evaluated against the + // substituted constant rather than skipped: null does not always mean false in a filter. + if (!missingKeyLocalPositions.isEmpty()) { + int[] missing = new int[missingKeyLocalPositions.size()]; + Object[] missingValues = new Object[missingKeyLocalPositions.size()]; + Object[] existenceDefaults = + ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema); + for (int i = 0; i < missing.length; i++) { + missing[i] = missingKeyLocalPositions.get(i); + missingValues[i] = existenceDefaults[keyIndices[missing[i]]]; + } + storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues); + + if (presentKeyColumns.isEmpty()) { + // All key columns missing: the rewritten predicate is fully constant. Evaluate once and + // apply uniformly to the whole file. + boolean keepAll = storageFilter.evalAllMissing(); + if (keepAll) { + // Predicate is constant-true for this file: no filtering to do. + storageFilter = null; + } else { + // Predicate is constant-false/null: no row from this file can pass the filter. + storageFilter = null; + hitEndOfData = true; + } + return; + } + } + + keyDescriptors = new ColumnDescriptor[presentKeyColumns.size()]; + keyRequired = new boolean[presentKeyColumns.size()]; + Types.MessageTypeBuilder keySchemaBuilder = Types.buildMessage(); + Set keyTopLevelNames = new HashSet<>(); + for (int i = 0; i < presentKeyColumns.size(); i++) { + ParquetColumn column = presentKeyColumns.get(i); + keyDescriptors[i] = column.descriptor().get(); + keyRequired[i] = column.required(); + // Preserve the field name/type as it appears at the top of requestedSchema. + String topLevelName = keyDescriptors[i].getPath()[0]; + keySchemaBuilder.addField(requestedSchema.getType(topLevelName)); + keyTopLevelNames.add(topLevelName); + } + keyOnlyRequestedSchema = keySchemaBuilder.named(requestedSchema.getName()); + + // Build the non-key (complement) schema. When the projection has at least one non-key column, + // phase 2 will switch lateMatReader's schema to this and read those columns under finalRanges. + // When the projection is *all* key columns (e.g. a scan whose only column is the bloom probe), + // splicing still pays off: phase 2 has nothing useful to read, so we skip it entirely (no read, + // no IO) and emit batches purely from the key queues. The `nonKeyRequestedSchema != null` check + // downstream gates phase-2 IO. + Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage(); + int nonKeyFieldCount = 0; + for (Type field : requestedSchema.getFields()) { + if (!keyTopLevelNames.contains(field.getName())) { + nonKeyBuilder.addField(field); + nonKeyFieldCount++; + } + } + if (nonKeyFieldCount > 0) { + nonKeyRequestedSchema = nonKeyBuilder.named(requestedSchema.getName()); + requireOffsetIndexesForPhase2(); + } + initializeSplicingState(presentKeyColumns); + + totalBlockCount = lateMatReader.getRowGroups().size(); + nextBlockIndex = 0; + } + + /** + * Fails now if any projected column of any row group lacks a Parquet offset index. + * + *

Phase 2 reads a strict subset of a row group's rows, which parquet can only do via the + * offset index; files written before parquet-mr 1.11, or by writers that omit it, have none. We + * cannot + * widen phase 2 to the whole block instead, because the key vectors already hold only the + * survivors and the batch would misalign -- and we cannot skip the filter either, since + * {@code extractStorageFilters} has already removed it from the post-scan Filter. + * + *

Every projected column is checked, not just the non-key ones phase 2 reads, because parquet + * builds one column index store per row group and reuses it. Phase 0 asks for the row ranges + * under the full requested schema, so {@code ColumnIndexStoreImpl.create} is called with the key + * columns in its path set, and it returns its {@code EMPTY} singleton as soon as any one of those + * paths has no offset index. {@code ParquetFileReader.getColumnIndexStore} memoizes that store + * per block, and {@code EMPTY.getOffsetIndex} throws for *every* column. So a key column with no + * offset index kills phase 2 too, with a raw {@code MissingOffsetIndexException} naming some + * non-key column and none of the guidance below. + * + *

Checking up front rather than at the first partially-kept row group is deliberate: whether + * phase 2 needs the offset index otherwise depends on how selective the filter turns out to be on + * this particular file, so the same query would fail or not depending on the data. This is + * conservative -- a filter that happens to keep every row of every block would not have needed + * the offset index -- but such a filter also saves nothing, so failing loudly loses nothing. + * + *

The check itself is free: {@code getOffsetIndexReference()} is a footer field that + * {@link #initialize} has already read. + */ + private void requireOffsetIndexesForPhase2() { + Set projectedPaths = new HashSet<>(); + for (ColumnDescriptor column : requestedSchema.getColumns()) { + projectedPaths.add(ColumnPath.get(column.getPath())); + } + List blocks = lateMatReader.getRowGroups(); + for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) { + for (ColumnChunkMetaData chunk : blocks.get(blockIdx).getColumns()) { + if (projectedPaths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) { + throw new IllegalStateException(String.format( + "Storage-filter pushdown requires a Parquet offset index to read a subset of a row " + + "group, but column %s of row group %d in %s has none. Set %s=false to read " + + "this file.", + chunk.getPath(), blockIdx, lateMatReader.getFile(), + SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key())); + } + } + } + } + + /** + * Populates splicing bookkeeping: {@link #isKeyTopLevel}, {@link #keyVectorQueues}, + * {@link #currentKeyAccumulators}. {@code keyColumnIndices} already index the top-level slots of + * {@link #sparkSchema} (same indexing as {@link #columnVectors}), so they map directly onto + * {@link #isKeyTopLevel}. After this method returns, the splicing path is fully initialized; + * {@link #nextBatch()} and {@link #close()} use {@link #isKeyTopLevel} as the active-state + * indicator. + */ + @SuppressWarnings("unchecked") + private void initializeSplicingState(List presentKeyColumns) { + int numTop = sparkSchema.fields().length; + isKeyTopLevel = new boolean[numTop]; + int[] keyIndices = storageFilter.keyColumnIndices(); + for (int slot : keyIndices) { + isKeyTopLevel[slot] = true; + } + int numKeys = presentKeyColumns.size(); + keyVectorQueues = new java.util.ArrayDeque[numKeys]; + for (int i = 0; i < numKeys; i++) { + keyVectorQueues[i] = new java.util.ArrayDeque<>(); + } + currentKeyAccumulators = new WritableColumnVector[numKeys]; + currentKeyAccumulatorRowCount = 0; + keyCopiers = new ValueCopier[numKeys]; + StructField[] fields = sparkRequestedSchema.fields(); + for (int i = 0; i < numKeys; i++) { + keyCopiers[i] = copierFor(fields[keyIndices[i]].dataType()); + } } /** @@ -478,6 +911,10 @@ private boolean containsPath(Type parquetType, String[] path, int depth) { private void checkEndOfRowGroup() throws IOException { if (rowsReturned != totalCountLoadedSoFar) return; + if (storageFilter != null) { + loadNextRowGroupWithLateMaterialization(); + return; + } PageReadStore pages = reader.readNextRowGroup(); if (pages == null) { throw new IOException("expecting more rows but reached last block. Read " @@ -492,6 +929,450 @@ private void checkEndOfRowGroup() throws IOException { totalCountLoadedSoFar += pages.getRowCount(); } + /** + * Loads the next row group using the three-phase late-materialization pattern, all driven by the + * single {@link #lateMatReader} with its requested schema mutated per phase: + * - Phase 0 (full schema): compute {@code pushedFilterRanges} from the pushed data filter via + * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}. + * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges}, + * evaluate the storage filter per row, build {@code finalRanges}. + * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}. + * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection). + * + * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). + * Sets {@link #hitEndOfData} when all row groups have been processed. + */ + private void loadNextRowGroupWithLateMaterialization() throws IOException { + while (nextBlockIndex < totalBlockCount) { + int blockIdx = nextBlockIndex++; + long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount(); + if (blockRowCount == 0) { + // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and + // trip parquet's own `from <= to` assertion. The plain read path skips them too. + continue; + } + + // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the + // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and + // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes + // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index + // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's + // ranges under the full schema, and with it off we do not call getRowRanges at all. + lateMatReader.setRequestedSchema(requestedSchema); + // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT + // options.useColumnIndexFilter(), so calling it unconditionally would keep applying + // column-index filtering after a user turned it off. That conf is the documented escape hatch + // for files whose column index is wrong, and trusting a wrong column index here would drop + // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no + // longer holds this predicate. + RowRanges pushedFilterRanges = useColumnIndexFilter + ? lateMatReader.getRowRanges(blockIdx) + : RowRanges.createSingle(blockRowCount); + if (pushedFilterRanges.rowCount() == 0) { + // Pushed data filter rejects this block entirely via column index. Not a storage-filter + // skip, so we don't increment storage-filter metrics. + continue; + } + + // Both byte metrics are always wired in production (FileSourceScanLike creates all five + // whenever storageFilters is non-empty), so this only skips the work on the test-only path + // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so + // there is nothing here to avoid on the production path. + StorageFilterMetrics m = storageFilter.metrics(); + SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup(); + SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering(); + boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null; + long baselineRows = pushedFilterRanges.rowCount(); + long baselineBytes = needBytes + ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema, + pushedFilterRanges) + : 0L; + + // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the + // storage filter per row. + lateMatReader.setRequestedSchema(keyOnlyRequestedSchema); + long phase1Bytes = needBytes + ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema, + pushedFilterRanges) + : 0L; + PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges); + if (keyPages == null) { + // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already + // know pushedFilterRanges selects at least one row. Skipping the block here would drop its + // surviving rows from the output, so assert rather than `continue`. + throw new IllegalStateException( + "No key pages for row group " + blockIdx + " despite " + + pushedFilterRanges.rowCount() + " rows selected by the pushed filter"); + } + RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges); + + if (finalRanges.rowCount() == 0) { + // Every surviving row was rejected by the storage filter; skip block entirely. We still + // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read + // are baseline - phase1 (the non-key bytes the no-filter path would have read). + SQLMetric rgSkipped = m.rowGroupsSkipped(); + if (rgSkipped != null) rgSkipped.add(1L); + SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup(); + if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows); + if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes); + continue; + } + + // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely + // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each + // batch from the key queues alone. + long keptRows; + long phase2Bytes; + PageReadStore dataPages = null; + if (nonKeyRequestedSchema == null) { + keptRows = finalRanges.rowCount(); + phase2Bytes = 0L; + } else { + lateMatReader.setRequestedSchema(nonKeyRequestedSchema); + // requireOffsetIndexesForPhase2() already established that every projected column of every + // row group has an offset index, so this page-filtering read cannot fail for want of one. + dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); + if (dataPages == null) { + // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges, + // both excluded above. Match phase 1 and fail with a message rather than an NPE. + throw new IllegalStateException( + "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount() + + " surviving rows"); + } + keptRows = dataPages.getRowCount(); + phase2Bytes = bytesAvoidedPf != null + ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema, + finalRanges) + : 0L; + } + long filteredRows = baselineRows - keptRows; + SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup(); + if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows); + if (bytesAvoidedPf != null) { + bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes); + } + + if (dataPages != null) { + if (rowIndexGenerator != null) { + rowIndexGenerator.initFromPageReadStore(dataPages); + } + for (int i = 0; i < columnVectors.length; i++) { + if (isKeyTopLevel[i]) { + // Key columns are sourced from the queues during emit; skip phase-2 reader init. + continue; + } + initColumnReader(dataPages, columnVectors[i]); + } + } + totalCountLoadedSoFar += keptRows; + return; + } + hitEndOfData = true; + } + + /** + * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads + * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are + * included, because both are read whenever any page of a chunk is read. + * + *

Two sources, chosen so this never causes IO of its own: + *

    + *
  • {@code rowRanges} covers the whole block: the answer is the sum of the chunks' + * {@code getTotalSize()}, which is already in the footer. This is the case that matters -- + * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges} + * is necessarily the whole block, because a narrower range can only come from column-index + * filtering, which builds the store as a side effect. + *
  • {@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path + * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store + * is guaranteed to exist here, so the walk is pure metadata arithmetic. + *
+ * + *

Columns absent from this physical file (schema evolution) contribute nothing, which is + * correct: the reader transfers nothing for them. Every caller for a given block walks the same + * metadata, so a skipped column drops out of the baseline and the per-phase totals alike. + */ + private static long compressedBytesForRowRanges( + ParquetFileReader reader, + int blockIndex, + MessageType schema, + RowRanges rowRanges) { + if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) { + return 0L; + } + BlockMetaData block = reader.getRowGroups().get(blockIndex); + long blockRowCount = block.getRowCount(); + Map chunks = new HashMap<>(); + for (ColumnChunkMetaData chunk : block.getColumns()) { + chunks.put(chunk.getPath(), chunk); + } + boolean wholeBlock = rowRanges.rowCount() == blockRowCount; + ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex); + long total = 0L; + for (ColumnDescriptor column : schema.getColumns()) { + ColumnPath path = ColumnPath.get(column.getPath()); + ColumnChunkMetaData chunk = chunks.get(path); + if (chunk == null) { + // Column is in the (clipped) requested schema but not in this file. + continue; + } + if (wholeBlock) { + total += chunk.getTotalSize(); + continue; + } + OffsetIndex offsetIndex; + try { + offsetIndex = ciStore.getOffsetIndex(path); + } catch (MissingOffsetIndexException e) { + continue; + } + if (offsetIndex == null) { + continue; + } + // The dictionary page is read whenever any data page of the chunk is, so count it here the + // same way parquet's ColumnIndexFilterUtils.calculateOffsetRanges does. + total += dictionaryPageSize(chunk); + int pageCount = offsetIndex.getPageCount(); + for (int i = 0; i < pageCount; i++) { + long from = offsetIndex.getFirstRowIndex(i); + long to = offsetIndex.getLastRowIndex(i, blockRowCount); + if (rowRanges.isOverlapping(from, to)) { + total += offsetIndex.getCompressedPageSize(i); + } + } + } + return total; + } + + /** + * Compressed size of a chunk's dictionary page, or 0 if it has none. + * {@link ColumnChunkMetaData#getStartingPos()} already resolves to the dictionary page offset + * when there is a valid one, so the gap up to the first data page is exactly the dictionary page. + */ + private static long dictionaryPageSize(ColumnChunkMetaData chunk) { + long startingPos = chunk.getStartingPos(); + long firstDataPageOffset = chunk.getFirstDataPageOffset(); + return startingPos < firstDataPageOffset ? firstDataPageOffset - startingPos : 0L; + } + + /** + * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in + * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row, + * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends + * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}). + * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a + * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is + * pushed too. + * + *

The result is a subset of {@code pushedFilterRanges}: rows not in {@code + * pushedFilterRanges} were never read and are implicitly excluded. + */ + private RowRanges evaluateStorageFilter( + PageReadStore keyPages, + RowRanges pushedFilterRanges) throws IOException { + ensureKeyScratchAllocated(); + VectorizedColumnReader[] readers = new VectorizedColumnReader[keyDescriptors.length]; + for (int i = 0; i < readers.length; i++) { + readers[i] = new VectorizedColumnReader( + keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode, + datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion); + } + ensureCurrentKeyAccumulatorsAllocated(); + + long keyRowsTotal = pushedFilterRanges.rowCount(); + PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator(); + RowRanges.Builder finalRangesBuilder = RowRanges.builder(); + long remaining = keyRowsTotal; + while (remaining > 0) { + int num = (int) Math.min((long) capacity, remaining); + for (int i = 0; i < keyScratchVectors.length; i++) { + keyScratchVectors[i].reset(); + readers[i].readBatch(num, keyScratchVectors[i], null, null); + } + keyScratchBatch.setNumRows(num); + for (int r = 0; r < num; r++) { + long blockRow = rowIndexIter.nextLong(); + if (storageFilter.test(keyScratchBatch.getRow(r))) { + finalRangesBuilder.addSelectedRow(blockRow); + appendSurvivorRowToAccumulators(r); + } + } + remaining -= num; + } + + finalizePartialAccumulators(); + + return finalRangesBuilder.build(); + } + + private void ensureKeyScratchAllocated() { + if (keyScratchVectors != null) return; + keyScratchVectors = new WritableColumnVector[keyDescriptors.length]; + boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP; + int[] keyIndices = storageFilter.keyColumnIndices(); + for (int i = 0; i < keyDescriptors.length; i++) { + DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType(); + keyScratchVectors[i] = useOffHeap + ? new OffHeapColumnVector(capacity, dt) + : new OnHeapColumnVector(capacity, dt); + } + keyScratchBatch = new ColumnarBatch(keyScratchVectors); + } + + /** + * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous + * accumulator was just pushed to the queue or this is the first row group). Each accumulator has + * {@link #capacity} rows. + */ + private void ensureCurrentKeyAccumulatorsAllocated() { + boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP; + int[] keyIndices = storageFilter.keyColumnIndices(); + for (int i = 0; i < currentKeyAccumulators.length; i++) { + if (currentKeyAccumulators[i] == null) { + DataType dt = sparkRequestedSchema.fields()[keyIndices[i]].dataType(); + currentKeyAccumulators[i] = useOffHeap + ? new OffHeapColumnVector(capacity, dt) + : new OnHeapColumnVector(capacity, dt); + } + } + currentKeyAccumulatorRowCount = 0; + } + + /** + * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding + * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues + * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay + * aligned. + */ + private void appendSurvivorRowToAccumulators(int srcRow) { + final int dstRow = currentKeyAccumulatorRowCount; + final WritableColumnVector[] accs = currentKeyAccumulators; + final WritableColumnVector[] srcs = keyScratchVectors; + final ValueCopier[] copiers = keyCopiers; + for (int i = 0, n = accs.length; i < n; i++) { + WritableColumnVector src = srcs[i]; + WritableColumnVector dst = accs[i]; + if (src.isNullAt(srcRow)) { + dst.putNull(dstRow); + } else { + copiers[i].copy(dst, dstRow, src, srcRow); + } + } + currentKeyAccumulatorRowCount = dstRow + 1; + if (currentKeyAccumulatorRowCount == capacity) { + for (int i = 0; i < currentKeyAccumulators.length; i++) { + keyVectorQueues[i].addLast(currentKeyAccumulators[i]); + currentKeyAccumulators[i] = null; + } + ensureCurrentKeyAccumulatorsAllocated(); + } + } + + /** + * Pushes any partially-filled accumulator into its queue at row-group end so the emit path can + * dequeue it as the row group's final batch. + */ + private void finalizePartialAccumulators() { + if (currentKeyAccumulatorRowCount == 0) return; + for (int i = 0; i < currentKeyAccumulators.length; i++) { + keyVectorQueues[i].addLast(currentKeyAccumulators[i]); + currentKeyAccumulators[i] = null; + } + currentKeyAccumulatorRowCount = 0; + } + + /** + * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over, + * any vectors still queued (e.g. on early termination), and partially-filled accumulators. + * Called from {@link #close()}. + */ + private void closeSplicingState() { + if (pendingCloseKeyVectors != null) { + for (WritableColumnVector v : pendingCloseKeyVectors) { + if (v != null) v.close(); + } + pendingCloseKeyVectors = null; + } + if (keyVectorQueues != null) { + for (java.util.ArrayDeque q : keyVectorQueues) { + if (q != null) { + for (WritableColumnVector v : q) v.close(); + q.clear(); + } + } + } + if (currentKeyAccumulators != null) { + for (WritableColumnVector v : currentKeyAccumulators) { + if (v != null) v.close(); + } + currentKeyAccumulators = null; + } + } + + /** + * Per-key-column value copier: appends one value from {@code src[srcRow]} to + * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving + * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources. + */ + @FunctionalInterface + private interface ValueCopier { + void copy(WritableColumnVector dst, int dstRow, WritableColumnVector src, int srcRow); + } + + /** + * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here + * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both + * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is + * unreachable. Teach both sides at once when adding a type; a type admitted there but missing + * here becomes a task failure instead of a planning-time rejection. + */ + private static ValueCopier copierFor(DataType dt) { + if (dt instanceof BooleanType) { + return (dst, dRow, src, sRow) -> dst.putBoolean(dRow, src.getBoolean(sRow)); + } + if (dt instanceof ByteType) { + return (dst, dRow, src, sRow) -> dst.putByte(dRow, src.getByte(sRow)); + } + if (dt instanceof ShortType) { + return (dst, dRow, src, sRow) -> dst.putShort(dRow, src.getShort(sRow)); + } + if (dt instanceof IntegerType + || dt instanceof DateType + || dt instanceof YearMonthIntervalType) { + return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow)); + } + if (dt instanceof LongType + || dt instanceof TimestampType + || dt instanceof TimestampNTZType + || dt instanceof TimeType + || dt instanceof DayTimeIntervalType) { + return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow)); + } + if (dt instanceof FloatType) { + return (dst, dRow, src, sRow) -> dst.putFloat(dRow, src.getFloat(sRow)); + } + if (dt instanceof DoubleType) { + return (dst, dRow, src, sRow) -> dst.putDouble(dRow, src.getDouble(sRow)); + } + if (dt instanceof DecimalType decimalType) { + int precision = decimalType.precision(); + if (precision <= Decimal.MAX_INT_DIGITS()) { + return (dst, dRow, src, sRow) -> dst.putInt(dRow, src.getInt(sRow)); + } + if (precision <= Decimal.MAX_LONG_DIGITS()) { + return (dst, dRow, src, sRow) -> dst.putLong(dRow, src.getLong(sRow)); + } + return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow)); + } + if (dt instanceof StringType + || dt instanceof VarcharType + || dt instanceof CharType + || dt instanceof BinaryType) { + return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow)); + } + throw new UnsupportedOperationException( + "Splicing storage-filter pushdown does not support key type: " + dt); + } + private void initColumnReader(PageReadStore pages, ParquetColumnVector cv) throws IOException { if (!missingColumns.contains(cv.getColumn())) { if (cv.getColumn().isPrimitive()) { @@ -518,20 +1399,28 @@ private void initColumnReader(PageReadStore pages, ParquetColumnVector cv) throw * use `OnHeapColumnVector` when `useOffHeap` is false, the constant columns * always use `ConstantColumnVector`. * + *

Data slots whose indices appear in {@code skipDataSlots} are left null. The splicing + * late-materialization path uses this to skip allocation for storage-filter key columns: those + * slots are sourced from per-key-column queues populated in phase 1 (see {@code + * nextBatchSplicing}). + * * Capacity is the initial capacity of the vector, and it will grow as necessary. * Capacity is in number of elements, not number of bytes. */ private ColumnVector[] allocateColumns( - int capacity, StructType schema, boolean useOffHeap, int constantColumnLength) { + int capacity, StructType schema, boolean useOffHeap, int constantColumnLength, + Set skipDataSlots) { StructField[] fields = schema.fields(); int fieldsLength = fields.length; ColumnVector[] vectors = new ColumnVector[fieldsLength]; if (useOffHeap) { for (int i = 0; i < fieldsLength - constantColumnLength; i++) { + if (skipDataSlots != null && skipDataSlots.contains(i)) continue; vectors[i] = new OffHeapColumnVector(capacity, fields[i].dataType()); } } else { for (int i = 0; i < fieldsLength - constantColumnLength; i++) { + if (skipDataSlots != null && skipDataSlots.contains(i)) continue; vectors[i] = new OnHeapColumnVector(capacity, fields[i].dataType()); } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index a727ccf565063..67a76449bfb95 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -303,6 +303,12 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { // Filters on non-partition columns. def dataFilters: Seq[Expression] + // Filters that should be evaluated lazily by the storage layer (e.g. parquet reader) for IO + // pruning of value columns based on key column evaluation. These may reference subqueries (e.g. a + // runtime bloom filter built from a join build side) and are materialized at task launch time. + // Defaults to Nil so that a scan which does not support storage-filter pushdown need not know + // about it. + def storageFilters: Seq[Expression] = Nil // Disable bucketed scan based on physical query plan, see rule // [[DisableUnnecessaryBucketedScan]] for details. def disableBucketedScan: Boolean @@ -555,7 +561,16 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { "PartitionFilters" -> seqToString(partitionFilters), "PushedFilters" -> seqToString(pushedFiltersForDisplay), "DataFilters" -> seqToString(dataFilters), - "Location" -> locationDesc) + "Location" -> locationDesc) ++ + // Only surface storage filters when the scan actually has some. `simpleString` renders every + // metadata entry verbatim, unlike `verboseStringWithOperatorId` which drops empty ones, so an + // unconditional entry would append `StorageFilters: []` to every file-scan explain line for a + // feature that is off by default. + (if (storageFilters.nonEmpty) { + Map("StorageFilters" -> seqToString(storageFilters)) + } else { + Map.empty[String, String] + }) relation.bucketSpec.map { spec => val bucketedKey = "Bucketed" @@ -642,7 +657,27 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { } else { None } - } ++ driverMetrics + } ++ storageFilterMetrics ++ driverMetrics + + protected lazy val storageFilterMetrics: Map[String, SQLMetric] = if (storageFilters.nonEmpty) { + // A row group is skipped, a row is excluded, a byte is avoided. See StorageFilterMetrics for + // why each counter uses its own verb. + Map( + FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED -> + SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"), + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP -> + SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (whole row group)"), + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP -> + SQLMetrics.createMetric(sparkContext, "rows excluded by storage filter (within row group)"), + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP -> + SQLMetrics.createSizeMetric(sparkContext, + "bytes avoided by storage filter (whole row group)"), + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING -> + SQLMetrics.createSizeMetric(sparkContext, + "bytes avoided by storage filter (page filtering)")) + } else { + Map.empty + } /** * A file listing that represents a file list as an array of [[PartitionDirectory]]. This extends @@ -702,6 +737,14 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { } } +object FileSourceScanLike { + val STORAGE_FILTER_ROW_GROUPS_SKIPPED = "storageFilterRowGroupsSkipped" + val STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP = "storageFilterRowsExcludedByRowGroup" + val STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP = "storageFilterRowsExcludedWithinRowGroup" + val STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP = "storageFilterBytesAvoidedByRowGroup" + val STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING = "storageFilterBytesAvoidedByPageFiltering" +} + /** * Physical plan node for scanning data from HadoopFsRelations. * @@ -715,6 +758,9 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { * @param tableIdentifier Identifier for the table in the metastore. * @param disableBucketedScan Disable bucketed scan based on physical query plan, see rule * [[DisableUnnecessaryBucketedScan]] for details. + * @param storageFilters Filters evaluated by the storage layer (e.g. parquet reader) to drive + * value-column IO pruning based on key-column evaluation. May contain + * subqueries materialized at task launch. */ case class FileSourceScanExec( @transient override val relation: HadoopFsRelation, @@ -727,7 +773,8 @@ case class FileSourceScanExec( override val dataFilters: Seq[Expression], override val tableIdentifier: Option[TableIdentifier], override val disableBucketedScan: Boolean = false, - override val markedForSingleTaskExecution: Boolean = false) + override val markedForSingleTaskExecution: Boolean = false, + override val storageFilters: Seq[Expression] = Nil) extends FileSourceScanLike { // Note that some vals referring the file-based relation are lazy intentionally @@ -752,15 +799,33 @@ case class FileSourceScanExec( lazy val inputRDD: RDD[InternalRow] = { val options = relation.options + (FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString) + // Only route through the storage-filter entry point when there is something to push. A + // `FileFormat` subclass that customizes reading by overriding `buildReaderWithPartitionValues` + // -- the long-standing entry point -- would otherwise be bypassed on every query, because + // `ParquetFileFormat` overrides `buildReaderWithStorageFilters` with a full reader + // implementation that the subclass knows nothing about. val readFile: (PartitionedFile) => Iterator[InternalRow] = - relation.fileFormat.buildReaderWithPartitionValues( - sparkSession = relation.sparkSession, - dataSchema = relation.dataSchema, - partitionSchema = relation.partitionSchema, - requiredSchema = requiredSchema, - filters = pushedDownFilters, - options = options, - hadoopConf = getHadoopConf(relation.sparkSession, relation.options)) + if (preparedStorageFilters.isEmpty) { + relation.fileFormat.buildReaderWithPartitionValues( + sparkSession = relation.sparkSession, + dataSchema = relation.dataSchema, + partitionSchema = relation.partitionSchema, + requiredSchema = requiredSchema, + filters = pushedDownFilters, + options = options, + hadoopConf = getHadoopConf(relation.sparkSession, relation.options)) + } else { + relation.fileFormat.buildReaderWithStorageFilters( + sparkSession = relation.sparkSession, + dataSchema = relation.dataSchema, + partitionSchema = relation.partitionSchema, + requiredSchema = requiredSchema, + filters = pushedDownFilters, + storageFilters = preparedStorageFilters, + options = options, + hadoopConf = getHadoopConf(relation.sparkSession, relation.options), + storageFilterMetrics = storageFilterMetrics) + } val readRDD = if (bucketedScan) { createBucketedReadRDD(relation.bucketSpec.get, readFile, dynamicallySelectedPartitions) @@ -771,6 +836,33 @@ case class FileSourceScanExec( readRDD } + // Materialize scalar subqueries inside storage filters to literals and bind AttributeReferences + // to BoundReferences targeting positions in `requiredSchema`. Subqueries must have been prepared + // by SparkPlan before this is forced (same contract as `pushedDownFilters`). + @transient + protected lazy val preparedStorageFilters: Seq[Expression] = { + if (storageFilters.isEmpty) { + Nil + } else { + // Trust the planning-time decision: when [[FileSourceStrategy.extractStorageFilters]] moved a + // bloom filter into [[storageFilters]], it removed that conjunct from the post-scan Filter. + // Re-checking the conf here would silently drop the filter if the user toggled it off between + // planning and execution, producing wrong results. The conf only gates whether extraction + // happens at planning time. + // + // `output` is constructed by FileSourceStrategy as + // `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++ + // constantMetadataColumns` + // and `requiredSchema` is the StructType of the first two groups, so the first + // `requiredSchema.length` attributes of `output` correspond 1:1 to requiredSchema fields. + val requestedDataAttrs = output.take(requiredSchema.length) + storageFilters.map { expr => + val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral } + BindReferences.bindReference(subqueryReplaced, requestedDataAttrs) + } + } + } + override def inputRDDs(): Seq[RDD[InternalRow]] = { inputRDD :: Nil } @@ -971,7 +1063,8 @@ case class FileSourceScanExec( QueryPlan.normalizePredicates(dataFilters, output), None, disableBucketedScan, - markedForSingleTaskExecution) + markedForSingleTaskExecution, + QueryPlan.normalizePredicates(storageFilters, output)) } override def getStream: Option[SparkDataStream] = stream diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala index 42ed6d782e34b..6503af3636ddf 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala @@ -29,6 +29,7 @@ import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes import org.apache.spark.sql.errors.QueryExecutionErrors +import org.apache.spark.sql.execution.metric.SQLMetric import org.apache.spark.sql.internal.{SessionStateHelper, SQLConf} import org.apache.spark.sql.sources.Filter import org.apache.spark.sql.types._ @@ -165,6 +166,47 @@ trait FileFormat { } } + /** + * Like [[buildReaderWithPartitionValues]] but additionally accepts a sequence of storage filters: + * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based + * on key-column evaluation (e.g., late materialization with a runtime bloom filter). + * + * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no + * storage filter at all. File formats that support storage-filter pushdown (e.g., Parquet) should + * override this method. + * + * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than + * dropping it. The planner removes an extracted conjunct from the post-scan `Filter`, so a reader + * that ignores it returns rows the filter rejects. Every other layer of the feature fails loudly + * for the same reason. Unreachable today, because `FileSourceStrategy.extractStorageFilters` only + * extracts for `ParquetFileFormat` itself, but that keeps the invariant one method away from the + * code that relies on it. + * + * Scalar subqueries inside `storageFilters` are expected to have been materialized before this + * method is called, so that the returned reader can be safely serialized to executors. + * + * `storageFilterMetrics` is an optional map of SQL metrics the reader can update during execution + * (e.g. number of row groups skipped). The scan is expected to expose these metrics via its + * `metrics` field so they show up in the SQL UI. + */ + def buildReaderWithStorageFilters( + sparkSession: SparkSession, + dataSchema: StructType, + partitionSchema: StructType, + requiredSchema: StructType, + filters: Seq[Filter], + storageFilters: Seq[Expression], + options: Map[String, String], + hadoopConf: Configuration, + storageFilterMetrics: Map[String, SQLMetric] = Map.empty + ): PartitionedFile => Iterator[InternalRow] = { + require(storageFilters.isEmpty, + s"${getClass.getSimpleName} does not support storage-filter pushdown, but was given " + + storageFilters.mkString("[", ", ", "]")) + buildReaderWithPartitionValues( + sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf) + } + /** * Create a file metadata struct column containing fields supported by the given file format. */ diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala index e2427222d8ebc..a087bc64a4a4a 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala @@ -33,6 +33,7 @@ import org.apache.spark.sql.catalyst.trees.TreePattern.{PLAN_EXPRESSION, SCALAR_ import org.apache.spark.sql.catalyst.types.DataTypeUtils import org.apache.spark.sql.classic.Strategy import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan} +import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, ParquetStorageFilter} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{DoubleType, FloatType, StructType} import org.apache.spark.util.ArrayImplicits._ @@ -151,6 +152,63 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { } } + /** + * Splits `afterScanFilters` into bloom-filter conjuncts that can be pushed to the storage layer + * for late materialization (returned as the first element) and the remaining filters that stay as + * a post-scan FilterExec (returned as the second element). + * + * Eligibility (statically checked here so the runtime never silently loses the filter): + * - The storage-filter pushdown SQL conf is on. + * - The file format is exactly [[ParquetFileFormat]]. Subclasses are excluded on purpose: they + * may customize reading by overriding `buildReaderWithPartitionValues`, and attaching storage + * filters would route the scan through `ParquetFileFormat`'s own reader instead, silently + * dropping whatever the subclass does. + * - The vectorized reader is feasible for the schema the reader will actually see, i.e. + * `partitionSchema ++ outputDataSchema` -- the same schema `ParquetFileFormat.buildReader` + * derives `enableVectorizedReader` from. + * - The conjunct is a top-level [[BloomFilterMightContain]] (not nested under OR/NOT). + * - The conjunct is deterministic. `ParquetStorageFilter.test` evaluates the predicate without + * calling `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a + * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. No + * such bloom exists today, because the only producer is `InjectRuntimeFilter` and a join key + * is deterministic, but this gate should not depend on a distant rule. + * - The bloom's value-side references are projected data columns whose type the reader's value + * copier supports (see [[ParquetStorageFilter.isSupportedKeyType]]). + * + * If any condition fails, ALL bloom filters stay in the second element to preserve the existing + * fallback behavior. + */ + private def extractStorageFilters( + afterScanFilters: ExpressionSet, + fsRelation: HadoopFsRelation, + readDataColumns: Seq[Attribute], + outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = { + val sparkSession = fsRelation.sparkSession + val sqlConf = sparkSession.sessionState.conf + if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters) + if (fsRelation.fileFormat.getClass != classOf[ParquetFileFormat]) { + return (Nil, afterScanFilters) + } + // Mirror the runtime check for vectorized read feasibility. Storage filters drive late + // materialization in the vectorized parquet reader; without it, the scan would silently + // ignore them. + val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields) + if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) { + return (Nil, afterScanFilters) + } + + val dataAttrs = AttributeSet(readDataColumns) + val (eligible, rest) = afterScanFilters.partition { + case bloom: BloomFilterMightContain => + val refs = bloom.valueExpression.references + bloom.deterministic && refs.nonEmpty && refs.forall { a => + dataAttrs.contains(a) && ParquetStorageFilter.isSupportedKeyType(a.dataType) + } + case _ => false + } + (eligible.toSeq, ExpressionSet(rest)) + } + def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match { case ScanOperation(projects, stayUpFilters, filters, l @ LogicalRelationWithTable(fsRelation: HadoopFsRelation, table)) => @@ -218,6 +276,10 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { logInfo(log"Post-Scan Filters: ${MDC(POST_SCAN_FILTERS, afterScanFilters.simpleString(maxToStringFields))}") + // `filterAttributes` is deliberately computed from `afterScanFilters` *before* storage-filter + // extraction (which happens further down, once `outputDataSchema` is known), so a column + // referenced only by an extracted bloom filter is still part of `requiredAttributes` and + // survives projection pruning -- the reader needs to read it to evaluate the filter. val filterAttributes = AttributeSet(afterScanFilters ++ stayUpFilters) val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects val requiredAttributes = AttributeSet(requiredExpressions) @@ -295,6 +357,15 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { val outputDataSchema = (readDataColumns ++ generatedMetadataColumns).toStructType + // Extract bloom-filter conjuncts that can be pushed to the storage layer for late + // materialization. Eligible bloom filters become `storageFilters` on the scan and are dropped + // from the post-scan Filter (the late-mat path produces exact output). Ineligible ones stay + // in `afterScanFilters` as the existing fallback. This runs here, rather than next to + // `afterScanFilters`, because eligibility depends on `outputDataSchema` -- the schema the + // reader will actually see, and hence what its vectorized-read feasibility is decided from. + val (storageFilters, remainingAfterScanFilters) = extractStorageFilters( + afterScanFilters, fsRelation, readDataColumns, outputDataSchema) + // The output rows will be produced during file scan operation in three steps: // (1) File format reader populates a `Row` with `readDataColumns` and // `fileFormatReaderGeneratedMetadataColumns` @@ -339,7 +410,8 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { rebindFileSourceMetadataAttributesInFilters(expandedDataFilters), table.map(_.identifier), markedForSingleTaskExecution = - l.getTagValue(MarkSingleTaskExecution.markTag).getOrElse(false)) + l.getTagValue(MarkSingleTaskExecution.markTag).getOrElse(false), + storageFilters = storageFilters) // extra Project node: wrap flat metadata columns to a metadata struct val withMetadataProjections = metadataStructOpt.map { metadataStruct => @@ -359,7 +431,8 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { }.getOrElse(scan) // bottom-most filters are put in the left of the list. - val finalFilters = afterScanFilters.toSeq.reduceOption(expressions.And).toSeq ++ stayUpFilters + val finalFilters = + remainingAfterScanFilters.toSeq.reduceOption(expressions.And).toSeq ++ stayUpFilters val withFilter = finalFilters.foldLeft(withMetadataProjections)((plan, cond) => { execution.FilterExec(cond, plan) }) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index 2e1216aebca53..fa33f0d1bd307 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -48,8 +48,10 @@ import org.apache.spark.sql.catalyst.parser.LegacyTypeStringParser import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils, RebaseDateTime} import org.apache.spark.sql.errors.QueryExecutionErrors +import org.apache.spark.sql.execution.FileSourceScanLike import org.apache.spark.sql.execution.datasources._ import org.apache.spark.sql.execution.datasources.parquet.types.ops.ParquetTypeOps +import org.apache.spark.sql.execution.metric.SQLMetric import org.apache.spark.sql.execution.vectorized.{ConstantColumnVector, OffHeapColumnVector, OnHeapColumnVector} import org.apache.spark.sql.internal.{SessionStateHelper, SQLConf} import org.apache.spark.sql.internal.SQLConf._ @@ -190,6 +192,21 @@ class ParquetFileFormat filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = { + buildReaderWithStorageFilters( + sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf, + Map.empty) + } + + override def buildReaderWithStorageFilters( + sparkSession: SparkSession, + dataSchema: StructType, + partitionSchema: StructType, + requiredSchema: StructType, + filters: Seq[Filter], + storageFilters: Seq[Expression], + options: Map[String, String], + hadoopConf: Configuration, + storageFilterMetrics: Map[String, SQLMetric]): PartitionedFile => Iterator[InternalRow] = { val sqlConf = getSqlConf(sparkSession) setupHadoopConf(hadoopConf, sqlConf, requiredSchema) @@ -230,6 +247,41 @@ class ParquetFileFormat val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val archiveFormatEnabled = parquetOptions.archiveFormatEnabled + // A non-empty `storageFilters` means `FileSourceStrategy.extractStorageFilters` already removed + // those conjuncts from the post-scan Filter, so there is no longer anything else in the plan + // that would apply them. Quietly not installing them here would return extra rows, so anything + // that stops us from honoring them has to fail loudly instead. + // + // `enableVectorizedReader` is recomputed from the live session conf when the RDD is built, i.e. + // after planning, so flipping spark.sql.parquet.enableVectorizedReader (or the nested-column + // variant) between planning and execution lands here. + val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) { + None + } else if (!enableVectorizedReader) { + throw new IllegalStateException( + "Cannot honor storage filters " + storageFilters.mkString("[", ", ", "]") + + " because the " + + "vectorized Parquet reader is disabled for schema " + resultSchema.catalogString + ". " + + "The scan was planned with storage-filter pushdown, which requires the vectorized " + + "reader; a vectorized-reader conf was most likely changed after the query was planned. " + + s"Set ${SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key}=false and re-run the query.") + } else { + val metrics = StorageFilterMetrics( + rowGroupsSkipped = storageFilterMetrics.getOrElse( + FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED, null), + rowsExcludedByRowGroup = storageFilterMetrics.getOrElse( + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP, null), + rowsExcludedWithinRowGroup = storageFilterMetrics.getOrElse( + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP, null), + bytesAvoidedByRowGroup = storageFilterMetrics.getOrElse( + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, null), + bytesAvoidedByPageFiltering = storageFilterMetrics.getOrElse( + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null)) + // `create` requires every condition extractStorageFilters already pre-checked, so it throws + // rather than letting us drop the filter. + Some(ParquetStorageFilter.create(storageFilters, requiredSchema, metrics)) + } + // Should always be set by FileSourceScanExec creating this. // Check conf before checking option, to allow working around an issue by changing conf. val returningBatch = sqlConf.parquetVectorizedReaderEnabled && @@ -321,7 +373,7 @@ class ParquetFileFormat buildVectorizedIterator( hadoopAttemptContext, split, file.partitionValues, partitionSchema, convertTz, datetimeRebaseSpec, int96RebaseSpec, enableOffHeapColumnVector, returningBatch, - capacity, openedFooter, shouldCloseInputStream) + capacity, openedFooter, shouldCloseInputStream, storageFilterOpt) } else { logDebug(s"Falling back to parquet-mr") buildRowBasedIterator( @@ -366,7 +418,8 @@ class ParquetFileFormat returningBatch: Boolean, batchSize: Int, openedFooter: OpenedParquetFooter, - shouldCloseInputStream: AtomicBoolean): Iterator[InternalRow] = { + shouldCloseInputStream: AtomicBoolean, + storageFilter: Option[ParquetStorageFilter]): Iterator[InternalRow] = { // scalastyle:on argcount assert(openedFooter.inputStreamOpt.isPresent) val vectorizedReader = new VectorizedParquetRecordReader( @@ -377,6 +430,7 @@ class ParquetFileFormat int96RebaseSpec.timeZone, enableOffHeapColumnVector && TaskContext.get() != null, batchSize) + storageFilter.foreach(vectorizedReader.setStorageFilter) // SPARK-37089: We cannot register a task completion listener to close this iterator here // because downstream exec nodes have already registered their listeners. Since listeners // are executed in reverse order of registration, a listener registered here would close the diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala new file mode 100644 index 0000000000000..d37f815921f69 --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.parquet + +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BoundReference, Expression, Literal, Predicate} +import org.apache.spark.sql.execution.metric.SQLMetric +import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType} + +/** + * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. All counters + * are scoped to what the storage filter added on top of a no-storage-filter read of the same + * projection. All fields are nullable; a null field disables that metric. + * + * Each counter names its own quantity, so the three verbs are deliberate. A row group is *skipped*, + * meaning its data columns were never read, though phase 1 did read its key columns. A row is + * *excluded*, meaning it never reached the output. A byte is *avoided*, meaning it was never + * transferred. + * + * The row counters' suffix says *where* the row was excluded, not by which mechanism: a row inside + * a kept row group is read as part of its page and dropped during decode, so page filtering did not + * save it. On an all-keys projection there is no page filtering at all, and + * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped. + * + * - [[rowGroupsSkipped]] counts row groups whose data columns were never read. + * - [[rowsExcludedByRowGroup]] sums rows excluded by full row-group skips (per skipped block, the + * count of rows that survived the pushed data filter). + * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept. + * - [[bytesAvoidedByRowGroup]] sums `baseline - phase1` for skipped row groups (phase 1 still + * reads the key column on every block, so the savings are the non-key bytes the no-filter path + * would have read; zero on all-keys-projection scans). + * - [[bytesAvoidedByPageFiltering]] sums `baseline - phase1 - phase2` for kept row groups: the + * non-key bytes pruned by `finalRanges` page selection beyond what phase 1 already read. + */ +case class StorageFilterMetrics( + rowGroupsSkipped: SQLMetric = null, + rowsExcludedByRowGroup: SQLMetric = null, + rowsExcludedWithinRowGroup: SQLMetric = null, + bytesAvoidedByRowGroup: SQLMetric = null, + bytesAvoidedByPageFiltering: SQLMetric = null) + +/** + * A runtime filter that the vectorized Parquet reader uses to drive late materialization: read + * key-column pages first, evaluate this filter per row to decide which rows survive, and skip + * data-column pages that do not overlap any surviving row range. + * + * [[keyColumnIndices]] are indices into the scan's requested data schema identifying the leaf + * columns referenced by the filter. [[boundExpression]] has its references rewritten to + * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must + * evaluate it against rows whose fields correspond to those key columns in that order. + * + * [[metrics]] carries the optional SQL metrics the reader updates as it applies the filter (always + * non-negative under splicing: phase 1 reads only the key columns, phase 2 reads only non-key + * columns under surviving row ranges, so the avoided bytes are exactly the non-key bytes the + * no-filter path would have read but we didn't). + */ +class ParquetStorageFilter private ( + val keyColumnIndices: Array[Int], + val boundExpression: Expression, + val metrics: StorageFilterMetrics) extends Serializable { + + // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer + // construction to first use on the executor. + @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression) + + def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow) + + /** + * Returns a new filter with the [[BoundReference]]s at `missingKeyLocalPositions` (positions in + * the local key-row layout, i.e. indices into [[keyColumnIndices]]) replaced by the constant the + * reader will actually materialize for that column, and the remaining [[BoundReference]]s + * renumbered to index into the reduced key-row layout. [[keyColumnIndices]] on the returned + * filter contains only the present columns in their original relative order. SQL metrics are + * shared with `this`. + * + * Used when a key column is missing from the physical parquet file (schema evolution). The + * predicate has to be evaluated against the substituted constant rather than skipped, because a + * null does not always mean `false` in a filter -- a `Coalesce`-wrapped reference still produces + * a non-null result, and `XxHash64` is `nullable = false` and hashes a null input to its seed. + * + * `missingKeyValues(i)` is the internal-format value the reader produces for + * `missingKeyLocalPositions(i)`: the column's existence DEFAULT when it has one, else null. + * Passing the default matters for correctness -- `ParquetColumnVector` writes the existence + * default into the output vector for a missing column, so evaluating the predicate against null + * would filter on a value the scan never returns and could drop rows that match. + * + * If all key positions are missing, the returned filter's [[boundExpression]] contains no + * [[BoundReference]]s and can be evaluated against [[InternalRow.empty]] to obtain a constant + * truth value (see [[evalAllMissing]]). + */ + def rewriteForMissingKeys( + missingKeyLocalPositions: Array[Int], + missingKeyValues: Array[Any]): ParquetStorageFilter = { + require(missingKeyLocalPositions.length == missingKeyValues.length, + "missingKeyLocalPositions and missingKeyValues must have the same length") + val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap + val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains) + val newPosOf = presentPositions.zipWithIndex.toMap + val rewritten = boundExpression.transform { + case b: BoundReference if substitution.contains(b.ordinal) => + Literal(substitution(b.ordinal), b.dataType) + case b: BoundReference => BoundReference(newPosOf(b.ordinal), b.dataType, b.nullable) + } + val newKeyColumnIndices = presentPositions.map(keyColumnIndices(_)).toArray + new ParquetStorageFilter(newKeyColumnIndices, rewritten, metrics) + } + + /** + * When every key column is missing (i.e. [[keyColumnIndices]] is empty after + * [[rewriteForMissingKeys]]), the bound expression is fully constant. Evaluates it and returns + * `true` iff the predicate is literally true; a null or false result is interpreted as "drop + * every row" by the reader. + */ + def evalAllMissing(): Boolean = { + require(keyColumnIndices.isEmpty, "evalAllMissing only valid when all key columns are missing") + boundExpression.eval(InternalRow.empty) == true + } +} + +object ParquetStorageFilter { + + /** + * Builds a [[ParquetStorageFilter]] from the given storage-filter expressions, already bound to + * the scan's requested data schema (i.e. [[BoundReference]]s with ordinals in + * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must + * satisfy all of them to survive. + * + * Every condition below is a hard precondition rather than a soft rejection. By the time this is + * called, `FileSourceStrategy.extractStorageFilters` has removed these conjuncts from the + * post-scan `Filter`, so nothing left in the plan would apply them; returning a "no filter" + * result would silently return rows the filter rejects. `extractStorageFilters` pre-checks all of + * it, so a violation here is a planner bug and failing is the only safe response. + * + * Callers that have no storage filters must not call this at all. + */ + def create( + boundExpressions: Seq[Expression], + requestedSchema: StructType, + metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = { + require(boundExpressions.nonEmpty, + "storage filters must be non-empty; callers with nothing to push must not call create") + val expr = boundExpressions.reduce(And) + + // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice + // is still one key column) and sorted. + // + // `sorted` is load-bearing, not cosmetic. It is not needed to keep `keyColumnIndices` and the + // remapped BoundReferences consistent with each other -- both derive from this list, so any + // order would agree. It is needed by a third consumer that does NOT go through the remapping: + // `VectorizedParquetRecordReader.nextBatchSplicing` walks the output batch slots in ascending + // order and pulls the survivor queues in key-row-position order, so it pairs "the k-th smallest + // key slot" with "key-row position k". That pairing is the identity only while this list is + // ascending. It cannot recover the order itself, because the reader records which slots are + // keys in a boolean array (`isKeyTopLevel`) that does not preserve their position here. + val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted + require(originalOrdinals.nonEmpty, + s"storage filter $expr has no bound reference to a key column") + require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length), + s"storage filter $expr references ordinals ${originalOrdinals.mkString("[", ", ", "]")} " + + s"outside the ${requestedSchema.length} fields of ${requestedSchema.catalogString}") + val unsupported = originalOrdinals.map(requestedSchema.fields(_)) + .filterNot(field => isSupportedKeyType(field.dataType)) + require(unsupported.isEmpty, + "storage filter key columns must have a type the vectorized reader can copy, but " + + unsupported.map(f => s"${f.name} ${f.dataType.catalogString}").mkString(", ") + + " do not; see ParquetStorageFilter.isSupportedKeyType") + + val indexMap = originalOrdinals.zipWithIndex.toMap + val remapped = expr.transform { + case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable) + } + + new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics) + } + + /** + * Whether `dt` is usable as a storage-filter key column type. + * + * This is the single authority on key-type eligibility: + * `FileSourceStrategy.extractStorageFilters` consults it at planning time and [[create]] + * re-checks it, so the vectorized reader's per-type value copier + * (`VectorizedParquetRecordReader.copierFor`) is only ever asked for a type listed here. The two + * must stay in lockstep -- adding a type here without teaching `copierFor` about it turns a + * planning-time rejection into a task failure. + * + * Narrower than `AtomicType`, for two different reasons: + * - `VariantType` cannot be supported: its Parquet representation is a group, not a primitive + * leaf, so phase 1 has nothing flat to read it into. (It is unreachable anyway -- + * `HashExpression.checkInputDataTypes` rejects variant, so no bloom can be built on one.) + * - `GeometryType` and `GeographyType` could be supported -- both map to a primitive Parquet + * BINARY and both are handled by `WritableColumnVector.isArray`, so the existing byte-array + * copier would work -- but no bloom can currently reference them: `HashExpression`'s codegen + * type dispatch has no case for either, so hashing one fails at codegen. They are left out + * until something can actually produce such a filter. + */ + def isSupportedKeyType(dt: DataType): Boolean = dt match { + case _: BooleanType | _: ByteType | _: ShortType | _: IntegerType | _: LongType => true + case _: FloatType | _: DoubleType | _: DecimalType => true + case _: DateType | _: TimestampType | _: TimestampNTZType | _: TimeType => true + case _: YearMonthIntervalType | _: DayTimeIntervalType => true + // StringType also covers CharType and VarcharType, which extend it. + case _: StringType | _: BinaryType => true + case _ => false + } +} diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala index 525e485f0afae..eadab16688acf 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala @@ -447,7 +447,8 @@ class DataFrameJoinSuite extends SharedSparkSession } assert(broadcastExchanges.size == 1) val tables = broadcastExchanges.head.collect { - case FileSourceScanExec(_, _, _, _, _, _, _, _, Some(tableIdent), _, _) => tableIdent + case FileSourceScanExec(_, _, _, _, _, _, _, _, Some(tableIdent), _, _, _) => + tableIdent } assert(tables.size == 1) assert(tables.head === diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala index c402475eed6b7..1e0d8f30803a9 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala @@ -1540,7 +1540,7 @@ class SubquerySuite extends SharedSparkSession // need to execute the query before we can examine fs.inputRDDs() assert(stripAQEPlan(df.queryExecution.executedPlan) match { case WholeStageCodegenExec(ColumnarToRowExec(InputAdapter( - fs @ FileSourceScanExec(_, _, _, _, partitionFilters, _, _, _, _, _, _)))) => + fs @ FileSourceScanExec(_, _, _, _, partitionFilters, _, _, _, _, _, _, _)))) => partitionFilters.exists(ExecSubqueryExpression.hasSubquery) && fs.inputRDDs().forall( _.asInstanceOf[FileScanRDD].filePartitions.forall( diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala new file mode 100644 index 0000000000000..b0026bdbd7083 --- /dev/null +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala @@ -0,0 +1,1965 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources.parquet + +import java.io.{ByteArrayOutputStream, File} +import java.net.URI +import java.time.LocalTime +import java.util.concurrent.atomic.AtomicLong + +import scala.collection.mutable +import scala.jdk.CollectionConverters._ + +import org.apache.hadoop.conf.Configuration +import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem} +import org.apache.hadoop.mapreduce.Job +import org.apache.parquet.hadoop.{ParquetInputFormat, ParquetOutputFormat} + +import org.apache.spark.paths.SparkPath +import org.apache.spark.sql.{sources, QueryTest, Row, SparkSession} +import org.apache.spark.sql.catalyst.InternalRow +import org.apache.spark.sql.catalyst.expressions.{And, Attribute, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64} +import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter} +import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec} +import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile} +import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} +import org.apache.spark.sql.functions.col +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types._ +import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} +import org.apache.spark.util.sketch.BloomFilter + +/** + * Tests the late-materialization path of [[VectorizedParquetRecordReader]] driven by a + * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter + * into the reader, and asserts correctness + the two storage-filter metrics. + */ +class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { + import testImplicits._ + + // Writes a parquet file with the given rows and row-group size; returns the path. + private def writeParquetFile( + dir: File, + rows: Seq[(Long, String)], + rowGroupSize: Long = 1024L, + pageSize: Option[Long] = None): String = { + val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath + val writer = rows.toDF("k", "v") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) + // Dictionary encoding off keeps row-group sizing predictable. Note this does NOT disable the + // column index, despite what an earlier version of this comment claimed. + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + // A small page size gives each row group several pages per column, which is what lets + // column-index filtering produce a row range narrower than the whole row group. + pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size)) + writer.parquet(outDir) + val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")) + assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir") + files(0).getAbsolutePath + } + + // Collects all rows from a reader initialized with the given storage filter. + private def readAll( + filePath: String, + storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = { + val reader = new VectorizedParquetRecordReader(false, 4096) + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, java.util.Arrays.asList("k", "v")) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[(Long, String)]() + while (reader.nextBatch()) { + val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] + val n = batch.numRows() + val kVec = batch.column(0) + val vVec = batch.column(1) + var i = 0 + while (i < n) { + collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString)) + i += 1 + } + } + (collected.toSeq, reader) + } + + // Builds a `k >= threshold` storage filter bound to position 0. + private def keyAtLeastFilter( + threshold: Long, + metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = { + val expr = GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(threshold)) + val requested = StructType(Seq( + StructField("k", LongType, nullable = false), StructField("v", StringType, nullable = false))) + ParquetStorageFilter.create(Seq(expr), requested, metrics) + } + + // Writes a single-column (just `k`) parquet file for the supplied key type via Spark's + // {@code Encoder}. + private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder]( + dir: File, + keys: Seq[T], + rowGroupSize: Long = 1024L): String = { + val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath + spark.createDataset(keys).toDF("k") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .parquet(outDir) + val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")) + assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir") + files(0).getAbsolutePath + } + + // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function + // pulls one value at a time from the batch's key column. + private def readKeyOnlyAll[T]( + filePath: String, + storageFilter: ParquetStorageFilter, + extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T, + capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) = { + val reader = new VectorizedParquetRecordReader(false, capacity) + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, java.util.Arrays.asList("k")) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[T]() + while (reader.nextBatch()) { + val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] + val n = batch.numRows() + val kVec = batch.column(0) + var i = 0 + while (i < n) { + collected += extract(kVec, i) + i += 1 + } + } + (collected.toSeq, reader) + } + + // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the + // given key type. + private def keyOnlyAtLeastFilter( + threshold: Literal, + keyType: DataType, + metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = { + val expr = GreaterThanOrEqual(BoundReference(0, keyType, nullable = false), threshold) + val requested = StructType(Seq(StructField("k", keyType, nullable = false))) + ParquetStorageFilter.create(Seq(expr), requested, metrics) + } + + test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") { + withTempDir { dir => + // 40 rows, ~small row group => >= 2 row groups. + val rows = (1L to 40L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") + val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") + val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") + val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") + val filter = keyAtLeastFilter(1000L, StorageFilterMetrics( + rowGroupsSkipped = rgSkipped, + rowsExcludedByRowGroup = rowsExcludedRg, + rowsExcludedWithinRowGroup = rowsExcludedPf, + bytesAvoidedByRowGroup = bytesAvoidedRg, + bytesAvoidedByPageFiltering = bytesAvoidedPf)) + val (result, reader) = readAll(path, filter) + try { + assert(result.isEmpty, "filter rejects all rows; no rows should be emitted") + assert(rgSkipped.value > 0, + s"expected at least one row group skipped; got ${rgSkipped.value}") + assert(rowsExcludedRg.value > 0, + s"expected rows excluded by whole-rowgroup skip; got ${rowsExcludedRg.value}") + assert(rowsExcludedPf.value == 0, + s"no partial-row-group filtering expected; got ${rowsExcludedPf.value}") + // Schema is (k: Long, v: String). Skipping a row group avoids the v-column bytes the + // no-storage-filter path would have read; phase 1 still pays for k. So avoided > 0. + assert(bytesAvoidedRg.value > 0, + s"expected non-key bytes avoided by whole row groups; got ${bytesAvoidedRg.value}") + assert(bytesAvoidedPf.value == 0, + s"no page-filtering bytes expected when all groups skipped; got ${bytesAvoidedPf.value}") + } finally { + reader.close() + } + } + } + + test("all rows survive: no skipping and no filtering") { + withTempDir { dir => + val rows = (1L to 40L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") + val filter = keyAtLeastFilter(0L, StorageFilterMetrics( + rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf)) + val (result, reader) = readAll(path, filter) + try { + assert(result.toSet == rows.toSet, s"all rows should round-trip; got ${result.size} rows") + assert(rgSkipped.value == 0, s"nothing should be skipped; got ${rgSkipped.value}") + assert(rowsExcludedPf.value == 0, + s"nothing should be filtered; got ${rowsExcludedPf.value}") + } finally { + reader.close() + } + } + } + + test("mixed: some row groups skipped, others partially kept") { + withTempDir { dir => + // Many rows + small row groups => guaranteed multiple row groups. + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") + // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped. + val filter = keyAtLeastFilter(195L, StorageFilterMetrics( + rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf)) + val (result, reader) = readAll(path, filter) + try { + // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by + // our finalRanges) to skip rows within partial pages, so emitted rows == survivors. + val expected = rows.filter(_._1 >= 195L).toSet + assert(result.toSet == expected, + s"expected exact filtering; got ${result.map(_._1).sorted}, " + + s"expected ${expected.map(_._1).toSeq.sorted}") + assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}") + } finally { + reader.close() + } + } + } + + test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") { + // When the projected schema contains only the bloom key, phase 2 is skipped entirely + // (`nonKeyRequestedSchema == null` in the reader). All output rows come from the per-key-column + // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads + // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero: + // there are no non-key bytes to skip. + withTempDir { dir => + val keys = (1L to 200L) + val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L) + + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") + val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") + val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") + // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned. + val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics( + rowGroupsSkipped = rgSkipped, + rowsExcludedWithinRowGroup = rowsExcludedPf, + bytesAvoidedByRowGroup = bytesAvoidedRg, + bytesAvoidedByPageFiltering = bytesAvoidedPf)) + val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i)) + try { + val expected = keys.filter(_ >= 195L).toSet + assert(result.toSet == expected, + s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}") + assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}") + // For an all-keys projection, the no-storage-filter path would have read the same key + // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0. + assert(bytesAvoidedRg.value == 0, + s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedRg.value}") + assert(bytesAvoidedPf.value == 0, + s"expected no non-key bytes to avoid on all-keys projection; got ${bytesAvoidedPf.value}") + } finally { + reader.close() + } + } + } + + test("multi-batch emit: survivor count exceeds capacity") { + // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises: + // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail. + // - pendingCloseKeyVectors getting closed at the start of every subsequent emit. + // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times. + withTempDir { dir => + val keys = (1L to 100L) + // Big rowGroupSize so all 100 rows fit in one row group. + val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 64 * 1024L) + // Filter accepts every row so the queue is fully populated. + val filter = keyOnlyAtLeastFilter(Literal(0L), LongType) + val (result, reader) = + readKeyOnlyAll(path, filter, (vec, i) => vec.getLong(i), capacity = 16) + try { + assert(result == keys.toSeq, + s"expected all keys returned in order across multiple batches; got ${result.size} rows") + } finally { + reader.close() + } + } + } + + test("int key column: filter survivors round-trip through phase 1 accumulators") { + // Covers the IntegerType branch of ValueCopier. + withTempDir { dir => + val keys = (1 to 100) + val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L) + val filter = keyOnlyAtLeastFilter(Literal(90), IntegerType) + val (result, reader) = readKeyOnlyAll(path, filter, (vec, i) => vec.getInt(i)) + try { + assert(result.toSet == keys.filter(_ >= 90).toSet, + s"expected int keys >= 90; got ${result.sorted}") + } finally { + reader.close() + } + } + } + + test("string key column: filter survivors round-trip through phase 1 accumulators") { + // Covers the StringType branch of ValueCopier (variable-length byte copy via putByteArray). + withTempDir { dir => + val keys = (1 to 20).map(i => f"k$i%03d") + val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L) + val filter = keyOnlyAtLeastFilter( + Literal.create("k015", StringType), StringType) + val (result, reader) = + readKeyOnlyAll(path, filter, (vec, i) => vec.getUTF8String(i).toString) + try { + assert(result.toSet == keys.filter(_ >= "k015").toSet, + s"expected string keys >= 'k015'; got ${result.sorted}") + } finally { + reader.close() + } + } + } + + test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") { + // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and + // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection + // would silently return rows the filter excludes. create fails instead. + val requested = StructType(Seq( + StructField("k", LongType, nullable = false), + StructField("v", StringType, nullable = false))) + + // Nothing to push: the caller is supposed to check this before calling. + val empty = intercept[IllegalArgumentException] { + ParquetStorageFilter.create(Seq.empty, requested) + } + assert(empty.getMessage.contains("must be non-empty"), empty.getMessage) + + // Ordinal 5 is out of range for a two-field requested schema. + val outOfRange = intercept[IllegalArgumentException] { + ParquetStorageFilter.create( + Seq(GreaterThanOrEqual(BoundReference(5, LongType, nullable = false), Literal(0L))), + requested) + } + assert(outOfRange.getMessage.contains("outside the 2 fields"), outOfRange.getMessage) + + // No bound reference at all, so there is no key column to read in phase 1. + val noRefs = intercept[IllegalArgumentException] { + ParquetStorageFilter.create(Seq(GreaterThanOrEqual(Literal(1L), Literal(0L))), requested) + } + assert(noRefs.getMessage.contains("no bound reference"), noRefs.getMessage) + + // A key type the reader has no value copier for. + val variantSchema = StructType(Seq(StructField("k", VariantType, nullable = true))) + val badType = intercept[IllegalArgumentException] { + ParquetStorageFilter.create( + Seq(IsNull(BoundReference(0, VariantType, nullable = true))), variantSchema) + } + assert(badType.getMessage.contains("isSupportedKeyType"), badType.getMessage) + } + + // Serializes a [[BloomFilter]] to bytes suitable for a [[Literal]]. + private def bloomBytes(bf: BloomFilter): Array[Byte] = { + val out = new ByteArrayOutputStream() + bf.writeTo(out) + out.toByteArray + } + + // Computes `XxHash64(v)` using the same seed Spark uses for runtime bloom filters. + private def xxHash64(v: Any, dt: DataType): Long = { + new XxHash64(Seq(Literal(v, dt))).eval(InternalRow.empty).asInstanceOf[Long] + } + + test("rewriteForMissingKeys: regular equi-join bloom probes the null key's hash") { + // Regular equi-join: the runtime bloom is BloomFilterMightContain(bloom, XxHash64(key)). + // XxHash64 is a HashExpression, so it is nullable = false and hashes a null input to its SEED + // rather than producing null. Substituting Literal(null) therefore leaves a concrete probe for + // `xxHash64(null)`, and whether the file is kept depends on whether that hash is in the bloom. + // Both directions are asserted so the outcome does not hinge on a lucky bloom miss. + val nullKeyHash = xxHash64(null, LongType) + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + + def rewriteWithBloomContaining(hashes: Long*): ParquetStorageFilter = { + val bf = BloomFilter.create(10, 128) + hashes.foreach(bf.putLong) + val expr = BloomFilterMightContain( + Literal(bloomBytes(bf), BinaryType), + new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))) + val filter = ParquetStorageFilter.create(Seq(expr), requested) + filter.rewriteForMissingKeys(Array(0), Array(null)) + } + + val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType)) + assert(dropped.keyColumnIndices.isEmpty, + "all key positions are missing; keyColumnIndices should be empty") + assert(!dropped.evalAllMissing(), + "the null key's hash is not in the bloom, so evalAllMissing must return false " + + "(skip the file)") + + val kept = rewriteWithBloomContaining(nullKeyHash) + assert(kept.evalAllMissing(), + "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)") + } + + test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") { + // A missing column that has a non-null existence DEFAULT is materialized by + // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated + // against the default -- evaluating against null could skip a whole file whose rows all match. + val defaultValue = 7L + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val bf = BloomFilter.create(10, 128) + bf.putLong(xxHash64(defaultValue, LongType)) + val expr = BloomFilterMightContain( + Literal(bloomBytes(bf), BinaryType), + new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))) + val filter = ParquetStorageFilter.create(Seq(expr), requested) + + // Substituting the default keeps the file, because the default's hash is in the bloom. + assert(filter.rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(), + "substituting the existence default must probe the default's hash and keep the file") + // Substituting null instead would drop it -- the bug this guards against. + assert(!filter.rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(), + "sanity check: substituting null probes a different hash and would drop the file") + } + + test("rewriteForMissingKeys: null-safe equi-join bloom keeps the file") { + // Null-safe equi-join: ExtractEquiJoinKeys rewrites `a <=> b` so the join key becomes + // Coalesce(key, default). After rewriteForMissingKeys substitutes null for the + // BoundReference, the coalesce produces `default` (0L here) and the bloom probe checks + // whether the default's hash is in the bloom (which it is, matching what the creation side + // would have inserted for its own null rows). Expected result: keep the file. + val defaultHash = xxHash64(0L, LongType) + val bf = BloomFilter.create(10, 128) + bf.putLong(defaultHash) + val bloomLit = Literal(bloomBytes(bf), BinaryType) + val expr = BloomFilterMightContain( + bloomLit, + new XxHash64(Seq(Coalesce(Seq( + BoundReference(0, LongType, nullable = true), + Literal.default(LongType)))))) + + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(expr), requested) + + val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null)) + assert(rewritten.keyColumnIndices.isEmpty, + "all key positions are missing; keyColumnIndices should be empty") + assert(rewritten.evalAllMissing(), + "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true") + } + + test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") { + // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to + // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't + // run the predicate here, only the rewrite's structure. + val bf = BloomFilter.create(10, 128) + bf.putLong(xxHash64(1L, LongType)) + val bloomLit = Literal(bloomBytes(bf), BinaryType) + // Two conjuncts: one on ordinal 0 (missing), one on ordinal 1 (present). + val exprs = Seq( + BloomFilterMightContain( + bloomLit, new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))), + BloomFilterMightContain( + bloomLit, new XxHash64(Seq(BoundReference(1, LongType, nullable = true))))) + + val requested = StructType(Seq( + StructField("a", LongType, nullable = true), + StructField("b", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(exprs, requested) + assert(filter.keyColumnIndices.toSeq == Seq(0, 1)) + + val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null)) + assert(rewritten.keyColumnIndices.toSeq == Seq(1), + "only the present key column should remain in keyColumnIndices") + val ordinals = rewritten.boundExpression.collect { + case b: BoundReference => b.ordinal + } + assert(ordinals == Seq(0), + "the remaining BoundReference (for ordinal 1 in requested schema) should be " + + s"renumbered to local position 0; got $ordinals") + } + + // Returns the FileSourceScanExec for a parquet read, with `storageFilters` attached. Goes through + // Spark's normal planning/execution machinery (not the test-only reader init), so it exercises + // preparedStorageFilters' subquery materialization + bind, the SQL conf check, + // ParquetFileFormat.buildReaderWithStorageFilters, and metric propagation. + private def scanWithStorageFilter( + path: String, + keyName: String, + threshold: Long): FileSourceScanExec = { + val df = spark.read.parquet(path).select("k", "v") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec found in plan: $plan")) + val keyAttr = scan.output.find(_.name == keyName).getOrElse(fail(s"No $keyName in scan output")) + val expr = GreaterThanOrEqual(keyAttr, Literal(threshold)) + scan.copy(storageFilters = Seq(expr)) + } + + // Executes a SparkPlan that may produce columnar batches. When the plan supports columnar output + // (typical for parquet scans with WSCG enabled), Spark's planner normally inserts a + // ColumnarToRowExec; since these tests bypass the planner, we wrap manually. + private def executePlanCollect(plan: SparkPlan): Array[(Long, String)] = { + val rowPlan = if (plan.supportsColumnar) ColumnarToRowExec(plan) else plan + rowPlan.executeCollect().map(r => (r.getLong(0), r.getString(1))) + } + + test("end-to-end via FileSourceScanExec: conf on, filter applied, metrics populated") { + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val scan = scanWithStorageFilter(path, "k", threshold = 195L) + val collected = executePlanCollect(scan).toSet + val expected = rows.filter(_._1 >= 195L).toSet + assert(collected == expected, s"got ${collected.toSeq.sortBy(_._1)}; expected $expected") + + val rgSkipped = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) + assert(rgSkipped.value >= 1, + s"expected at least one row group skipped via storage filter; got ${rgSkipped.value}") + } + } + } + + test("pushed data filter on a non-key column + storage filter on key: cross-propagation works") { + // A pushed data filter on a non-key column and a storage filter on the key column have to + // compose: phase 0 derives its row ranges from the pushed filter, phase 1 narrows them with the + // storage filter, and phase 2 reads the non-key columns under the intersection. + // + // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes + // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is + // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq -- so a `!=` predicate here would be + // satisfied by a phase 0 that saw the wrong schema, and would prove nothing. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, f"v_$i%03d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val df = spark.read.parquet(path).select("k", "v").filter("v > 'v_000'") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + assert(scan.simpleString(200).contains("GreaterThan(v,"), + s"the data filter must actually be pushed for this test to mean anything: " + + scan.simpleString(200)) + // Storage filter on `k` (key column). + val keyAttr = scan.output.find(_.name == "k").get + val withSF = scan.copy( + storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(100L)))) + + val collected = executePlanCollect(withSF).toSet + // Every row satisfies v > 'v_000', so the storage filter alone decides the result. + val expected = rows.filter(_._1 >= 100L).toSet + assert(collected == expected, + s"got ${collected.toSeq.sortBy(_._1)}; expected ${expected.toSeq.sortBy(_._1)}") + } + } + } + + // ----- FileSourceStrategy bloom-filter extraction ----- + + // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan. + private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = { + plan.collect { + case f: FilterExec => + f.condition.collect { case _: BloomFilterMightContain => 1 }.sum + }.sum + } + + // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters. + private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = { + plan.collect { + case s: FileSourceScanExec => + s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum + }.sum + } + + // Sets up two parquet tables and runs a join that triggers `InjectRuntimeFilter` for the + // application-side scan. Returns the executed plan and the query result for inspection. Tables + // are cleaned up automatically by the calling test (table names are passed through). + private def runBloomFilterJoin(): (SparkPlan, Array[Row]) = { + val query = + """SELECT bf1.k, bf1.v + |FROM bf1 JOIN bf2 ON bf1.k = bf2.k + |WHERE bf2.v = 5 + |""".stripMargin + val df = spark.sql(query) + (df.queryExecution.executedPlan, df.collect()) + } + + // Creates two parquet tables (bf1: large, bf2: small with selective filter) for join tests. + private def withBloomFilterTables(body: => Unit): Unit = { + withTable("bf1", "bf2") { + // bf1 = "large" application side: 600 rows. + spark.range(600).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf1") + // bf2 = "small" creation side: 30 rows, with a selective filter (v = 5 keeps 1 row). + spark.range(30).selectExpr("id AS k", "id AS v").write.format("parquet").saveAsTable("bf2") + body + } + } + + test("FileSourceStrategy extracts bloom filter into scan.storageFilters when conf is on") { + withBloomFilterTables { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val (plan, _) = runBloomFilterJoin() + val storageBlooms = countBloomFiltersInStorageFilters(plan) + val postScanBlooms = countBloomFiltersInPostScanFilters(plan) + assert(storageBlooms >= 1, + s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" + + s"Plan:\n$plan") + assert(postScanBlooms == 0, + s"expected no bloom filter in any post-scan FilterExec; got $postScanBlooms.\n" + + s"Plan:\n$plan") + } + } + } + + test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") { + withBloomFilterTables { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val (plan, _) = runBloomFilterJoin() + val storageBlooms = countBloomFiltersInStorageFilters(plan) + val postScanBlooms = countBloomFiltersInPostScanFilters(plan) + assert(storageBlooms == 0, + s"expected no bloom on scan.storageFilters when conf is off; got $storageBlooms") + assert(postScanBlooms >= 1, + s"expected bloom in post-scan FilterExec when conf is off; got $postScanBlooms.\n" + + s"Plan:\n$plan") + } + } + } + + test("FileSourceStrategy extraction preserves query results") { + withBloomFilterTables { + val baseConf = Map( + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") + val resultConfOff = withSQLConf( + (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "false")).toSeq: _*) { + runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet + } + val resultConfOn = withSQLConf( + (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true")).toSeq: _*) { + runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet + } + assert(resultConfOn == resultConfOff, + s"results differ between conf-on and conf-off: on=$resultConfOn off=$resultConfOff") + } + } + + test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") { + // `ParquetStorageFilter.test` evaluates the predicate without calling + // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a + // `Nondeterministic` expression, so a non-deterministic conjunct has to stay behind. No + // producer builds one today, hence the hand-built plan: `InjectRuntimeFilter`'s blooms hash + // join keys, which are deterministic. + withTempDir { dir => + val rows = (1L to 50L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val bf = BloomFilter.create(10, 128) + bf.putLong(xxHash64(42L, LongType)) + val bloomLit = Literal(bloomBytes(bf), BinaryType) + val relation = spark.read.parquet(path).select("k", "v").queryExecution.optimizedPlan + val k = relation.output.find(_.name == "k").getOrElse(fail("no k in the relation output")) + + def extractedBlooms(valueExpr: Expression): (Int, Int) = { + val logical = LogicalFilter(BloomFilterMightContain(bloomLit, valueExpr), relation) + val physical = FileSourceStrategy(logical).headOption + .getOrElse(fail(s"FileSourceStrategy did not plan $logical")) + (countBloomFiltersInStorageFilters(physical), + countBloomFiltersInPostScanFilters(physical)) + } + + // Control: the same bloom over the key alone is extracted, so the arms differ in exactly + // one thing. + val (deterministicInScan, deterministicPostScan) = + extractedBlooms(new XxHash64(Seq(k))) + assert(deterministicInScan == 1 && deterministicPostScan == 0, + s"a deterministic bloom must be extracted; got scan=$deterministicInScan " + + s"postScan=$deterministicPostScan") + + // `Rand` contributes no reference, so `k` is still the only key column and only the + // determinism gate can reject this one. + val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L)))) + assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic") + val (inScan, postScan) = extractedBlooms(nonDeterministic) + assert(inScan == 0, s"a non-deterministic bloom must not be extracted; got $inScan") + assert(postScan == 1, s"it must stay in the post-scan Filter; got $postScan") + } + } + } + + // ----- Generic reader plumbing for the coverage tests below ----- + + // Writes a single-column (`k`) parquet file from a SQL expression over `id`, avoiding the need + // for an Encoder per key type. `keyExpr` is evaluated over `spark.range(1, n + 1)`. + private def writeKeyOnlyParquetFileFromSql( + dir: File, + keyExpr: String, + n: Long = 100L, + rowGroupSize: Long = 256L, + dictionary: Boolean = false): String = { + val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath + spark.range(1, n + 1).selectExpr(s"$keyExpr AS k") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString) + .parquet(outDir) + val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")) + assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir") + files(0).getAbsolutePath + } + + // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which + // selects the plain (non-splicing) vectorized path. + private def readAllWith[T]( + filePath: String, + columns: Seq[String], + storageFilter: ParquetStorageFilter, + extract: (ColumnarBatch, Int) => T, + capacity: Int = 4096, + useOffHeap: Boolean = false, + partitionColumns: StructType = new StructType(), + partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = { + val reader = new VectorizedParquetRecordReader(useOffHeap, capacity) + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, columns.asJava) + reader.initBatch(partitionColumns, partitionValues) + val collected = mutable.ArrayBuffer[T]() + while (reader.nextBatch()) { + val batch = reader.resultBatch() + var i = 0 + val n = batch.numRows() + while (i < n) { + collected += extract(batch, i) + i += 1 + } + } + (collected.toSeq, reader) + } + + // Renders one column value as a string using its *internal* representation, so the splicing path + // and the plain path can be compared without going through external type conversion. + private def renderValue(vec: ColumnVector, i: Int, dt: DataType): String = { + if (vec.isNullAt(i)) { + "null" + } else { + dt match { + case BooleanType => vec.getBoolean(i).toString + case ByteType => vec.getByte(i).toString + case ShortType => vec.getShort(i).toString + case IntegerType | DateType | _: YearMonthIntervalType => vec.getInt(i).toString + case LongType | TimestampType | TimestampNTZType | _: TimeType | + _: DayTimeIntervalType => vec.getLong(i).toString + case FloatType => vec.getFloat(i).toString + case DoubleType => vec.getDouble(i).toString + case d: DecimalType => vec.getDecimal(i, d.precision, d.scale).toString + case _: StringType => vec.getUTF8String(i).toString + case BinaryType => vec.getBinary(i).mkString(",") + case other => fail(s"renderValue does not handle $other") + } + } + } + + // Reads a key-only file through the plain (no storage filter) path and keeps the rows the given + // bound predicate accepts. This is the oracle for the splicing path: whatever the plain reader + // returns, filtered in Scala, is exactly what splicing must produce. + private def survivorsViaPlainPath( + filePath: String, + dt: DataType, + boundPredicate: org.apache.spark.sql.catalyst.expressions.Expression): Seq[String] = { + val predicate = Predicate.create(boundPredicate) + val (rows, reader) = readAllWith( + filePath, Seq("k"), null, + (b, i) => (renderValue(b.column(0), i, dt), b.getRow(i).copy())) + try { + rows.collect { case (rendered, row) if predicate.eval(row) => rendered } + } finally { + reader.close() + } + } + + // ----- Key-type coverage: one case per ValueCopier branch ----- + + // (case name, SQL expression producing `k`, Spark type, threshold as an external value) + private val keyTypeCases: Seq[(String, String, DataType, Any)] = Seq( + ("boolean", "id % 2 = 0", BooleanType, true), + ("byte", "CAST(id AS BYTE)", ByteType, 90.toByte), + ("short", "CAST(id AS SHORT)", ShortType, 90.toShort), + ("int", "CAST(id AS INT)", IntegerType, 90), + ("long", "id", LongType, 90L), + ("float", "CAST(id AS FLOAT)", FloatType, 90.0f), + ("double", "CAST(id AS DOUBLE)", DoubleType, 90.0d), + ("string", "LPAD(CAST(id AS STRING), 5, '0')", StringType, "00090"), + ("date", "DATE '2020-01-01' + CAST(id AS INT)", DateType, java.time.LocalDate.of(2020, 4, 1)), + ("timestamp", + "TIMESTAMPADD(SECOND, id, TIMESTAMP '2020-01-01 00:00:00')", + TimestampType, + java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30) + .atZone(java.time.ZoneId.systemDefault()).toInstant), + ("timestamp_ntz", + "TIMESTAMPADD(SECOND, id, TIMESTAMP_NTZ '2020-01-01 00:00:00')", + TimestampNTZType, + java.time.LocalDateTime.of(2020, 1, 1, 0, 1, 30)), + ("year_month_interval", "MAKE_YM_INTERVAL(0, CAST(id AS INT))", + YearMonthIntervalType(), java.time.Period.ofMonths(90)), + ("day_time_interval", "MAKE_DT_INTERVAL(0, 0, 0, CAST(id AS DOUBLE))", + DayTimeIntervalType(), java.time.Duration.ofSeconds(90)), + ("decimal_int", "CAST(id AS DECIMAL(9,2))", DecimalType(9, 2), BigDecimal("90.00")), + ("decimal_long", "CAST(id AS DECIMAL(18,2))", DecimalType(18, 2), BigDecimal("90.00")), + ("decimal_binary", "CAST(id AS DECIMAL(30,2))", DecimalType(30, 2), BigDecimal("90.00")), + ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType, + "00090".getBytes("UTF-8"))) + + // Run every key type both without and WITH dictionary encoding. Dictionary encoding is parquet's + // production default, and it is the case where the phase 1 scratch vectors carry a Dictionary + // plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's decode branch + // rather + // than straight out of the value array. + for { + (name, keyExpr, dt, threshold) <- keyTypeCases + dictionary <- Seq(false, true) + } { + val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded" + test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") { + // TIMESTAMP_MICROS rather than Spark's default INT96, which the reader only accepts with + // int96AsTimestamp and which is not the INT64 copier branch we want to cover here. + withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") { + withTempDir { dir => + val path = writeKeyOnlyParquetFileFromSql(dir, keyExpr, dictionary = dictionary) + val bound = GreaterThanOrEqual( + BoundReference(0, dt, nullable = true), Literal.create(threshold, dt)) + val expected = survivorsViaPlainPath(path, dt, bound) + assert(expected.nonEmpty && expected.size < 100, + s"the $name case should keep some but not all rows; kept ${expected.size}") + + val requested = StructType(Seq(StructField("k", dt, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + val (result, reader) = readAllWith( + path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt)) + try { + assert(result == expected, + s"splicing path disagrees with the plain path for $name ($encoding):\n" + + s" splicing: $result\n plain: $expected") + } finally { + reader.close() + } + } + } + } + } + + test("TIME key column is supported: isSupportedKeyType admits it and the copier handles it") { + // Regression test: TimeType is an AtomicType that passes every planning-time gate (it is + // batch-readable and XxHash64 hashes it, so InjectRuntimeFilter will build a bloom on a TIME + // join key), so a key-type whitelist that omitted it would fail the task at reader init. + assert(ParquetStorageFilter.isSupportedKeyType(TimeType(6)), + "TIME must be an eligible storage-filter key type") + withTempDir { dir => + val times = (1 to 100).map(i => LocalTime.ofSecondOfDay(i.toLong)) + val path = writeKeyOnlyParquetFile(dir, times, rowGroupSize = 256L) + val dt = TimeType(6) + val bound = GreaterThanOrEqual( + BoundReference(0, dt, nullable = true), + Literal.create(LocalTime.ofSecondOfDay(90L), dt)) + val expected = survivorsViaPlainPath(path, dt, bound) + assert(expected.size == 11, s"expected the last 11 of 100 TIME keys; got ${expected.size}") + + val requested = StructType(Seq(StructField("k", dt, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + val (result, reader) = + readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, dt)) + try { + assert(result == expected, s"got $result; expected $expected") + } finally { + reader.close() + } + } + } + + test("isSupportedKeyType covers exactly the types the reader can copy") { + // This is the contract that keeps FileSourceStrategy, ParquetStorageFilter.create and + // VectorizedParquetRecordReader.copierFor in lockstep. A type admitted here but missing from + // copierFor turns a planning-time rejection into a task failure. + val supported: Seq[DataType] = Seq( + BooleanType, ByteType, ShortType, IntegerType, LongType, FloatType, DoubleType, + DecimalType(9, 2), DecimalType(18, 2), DecimalType(30, 2), DateType, TimestampType, + TimestampNTZType, TimeType(6), YearMonthIntervalType(), DayTimeIntervalType(), + StringType, VarcharType(10), CharType(10), BinaryType) + supported.foreach { dt => + assert(ParquetStorageFilter.isSupportedKeyType(dt), s"$dt should be a supported key type") + } + // Atomic but with no primitive Parquet leaf to accumulate into, plus the non-atomic types. + val unsupported: Seq[DataType] = Seq( + VariantType, NullType, ArrayType(IntegerType), MapType(IntegerType, IntegerType), + new StructType().add("a", IntegerType)) + unsupported.foreach { dt => + assert(!ParquetStorageFilter.isSupportedKeyType(dt), + s"$dt should NOT be a supported key type") + } + } + + // ----- Null keys, multiple keys, partition columns, off-heap, row-at-a-time ----- + + test("nullable key column: surviving null keys are copied through as nulls") { + // The predicate deliberately accepts nulls, so appendSurvivorRowToAccumulators must take its + // dst.putNull branch. Every other test uses a non-nullable key, leaving that branch dead. + withTempDir { dir => + val path = writeKeyOnlyParquetFileFromSql( + dir, "CASE WHEN id % 10 = 0 THEN NULL ELSE id END") + val ref = BoundReference(0, LongType, nullable = true) + val bound = Or(IsNull(ref), GreaterThanOrEqual(ref, Literal(90L))) + val expected = survivorsViaPlainPath(path, LongType, bound) + assert(expected.count(_ == "null") == 10, s"expected 10 null keys; got $expected") + + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + val (result, reader) = + readAllWith(path, Seq("k"), filter, (b, i) => renderValue(b.column(0), i, LongType)) + try { + assert(result == expected, s"got $result; expected $expected") + } finally { + reader.close() + } + } + } + + test("two key columns: both accumulators stay aligned with each other and with the data column") { + withTempDir { dir => + val outDir = new File(dir, "twokeys").getAbsolutePath + spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, 256L) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .parquet(outDir) + val path = new File(outDir).listFiles((_, n) => n.endsWith(".parquet"))(0).getAbsolutePath + + // a >= 100 AND b <= 300 => a in [100, 150] + val bound = And( + GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L)), + LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L))) + val requested = StructType(Seq( + StructField("a", LongType, nullable = true), + StructField("b", LongType, nullable = true), + StructField("c", StringType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + assert(filter.keyColumnIndices.toSeq == Seq(0, 1), "both key columns should be recognized") + + val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter, + (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i), + b.column(2).getUTF8String(i).toString)) + try { + val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i")) + assert(result == expected, + s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})") + } finally { + reader.close() + } + } + } + + test("partition columns are preserved alongside spliced key columns") { + // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits + // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + val filter = keyAtLeastFilter(195L) + val partitionColumns = new StructType().add("p", IntegerType) + val (result, reader) = readAllWith( + path, Seq("k", "v"), filter, + (b, i) => (b.column(0).getLong(i), b.column(2).getInt(i)), + partitionColumns = partitionColumns, + partitionValues = InternalRow(7)) + try { + assert(result.map(_._1) == (195L to 200L), + s"expected keys 195..200; got ${result.map(_._1)}") + assert(result.forall(_._2 == 7), + s"every row should carry partition value 7; got ${result.map(_._2).distinct}") + } finally { + reader.close() + } + } + } + + Seq(false, true).foreach { useOffHeap => + val mode = if (useOffHeap) "off-heap" else "on-heap" + test(s"$mode vectors: multi-batch emit closes and reallocates survivor vectors correctly") { + // Off-heap is where the close/free hazards actually bite: OffHeapColumnVector.close() frees + // the native buffer, so a double close or a read after close is a crash rather than stale + // data. capacity = 16 over 100 survivors forces 7 emits, each closing the previous emit's + // dequeued key vectors. + withTempDir { dir => + val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 100L, rowGroupSize = 64 * 1024L) + val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(0L)) + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + val (result, reader) = readAllWith( + path, Seq("k"), filter, (b, i) => b.column(0).getLong(i), + capacity = 16, useOffHeap = useOffHeap) + try { + assert(result == (1L to 100L), s"expected all 100 keys in order; got ${result.size} rows") + } finally { + reader.close() + } + } + } + + test(s"$mode vectors: row-group skipping and page filtering over a mixed projection") { + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") + val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") + val filter = keyAtLeastFilter(195L, StorageFilterMetrics( + bytesAvoidedByRowGroup = bytesAvoidedRg, + bytesAvoidedByPageFiltering = bytesAvoidedPf)) + val reader = new VectorizedParquetRecordReader(useOffHeap, 4096) + reader.setStorageFilter(filter) + reader.initialize(path, Seq("k", "v").asJava) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[(Long, String)]() + try { + while (reader.nextBatch()) { + val batch = reader.resultBatch() + var i = 0 + while (i < batch.numRows()) { + collected += ((batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString)) + i += 1 + } + } + assert(collected.toSeq == rows.filter(_._1 >= 195L), + s"expected exact filtering; got ${collected.map(_._1)}") + // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the + // partially kept one. Both counters must be non-negative, and together positive. + assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0, + s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " + + s"pf=${bytesAvoidedPf.value}") + assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0, + "a mixed projection with skipped row groups should avoid some non-key bytes") + } finally { + reader.close() + } + } + } + } + + test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") { + // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an + // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with + // a capacity small enough to span several batches. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + val filter = keyAtLeastFilter(100L) + val reader = new VectorizedParquetRecordReader(false, 8) + try { + reader.setStorageFilter(filter) + reader.initialize(path, Seq("k", "v").asJava) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[(Long, String)]() + while (reader.nextKeyValue()) { + val row = reader.getCurrentValue().asInstanceOf[InternalRow] + collected += ((row.getLong(0), row.getString(1))) + } + assert(collected.toSeq == rows.filter(_._1 >= 100L), + s"expected keys 100..200 row by row; got ${collected.size} rows") + } finally { + reader.close() + } + } + } + + // ----- Schema evolution: a column missing from the physical file ----- + + test("non-key column missing from a file: byte-avoided metrics tolerate a missing offset index") { + // ColumnIndexStore returns a null OffsetIndex for a column that is in the (clipped) requested + // schema but absent from the row group, which is exactly what schema evolution produces. The + // avoided-bytes walk runs over the full requested schema on every row group of every file, so + // without a null guard this is an NPE on the first row group of the older file. + withTempDir { dir => + val base = new File(dir, "merged").getAbsolutePath + // Older file: (k, v). Newer file: (k, v, w). + (1L to 200L).map(i => (i, s"v_$i")).toDF("k", "v") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, 256L) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .mode("append") + .parquet(base) + (201L to 400L).map(i => (i, s"v_$i", i * 2)).toDF("k", "v", "w") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, 256L) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .mode("append") + .parquet(base) + + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val df = spark.read.option("mergeSchema", "true").parquet(base).select("k", "v", "w") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output")) + // Keeps rows from BOTH files, so the older one (where `w` is missing) is really read, and + // drops the leading row groups of the older file so the skip path is exercised there too. + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L)))) + + val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF + val collected = rowPlan.executeCollect() + .map(r => (r.getLong(0), r.getString(1), if (r.isNullAt(2)) None else Some(r.getLong(2)))) + .toSet + val expected = ((150L to 200L).map(i => (i, s"v_$i", None)) ++ + (201L to 400L).map(i => (i, s"v_$i", Some(i * 2)))).toSet + assert(collected == expected, + s"expected ${expected.size} rows across both schemas; got ${collected.size}") + + // The avoided-bytes counters are what walk the offset index of the missing column. Their + // being populated and non-negative is the evidence that the walk ran and coped. + val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) + val bytesPf = + withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING) + assert(bytesRg.value >= 0 && bytesPf.value >= 0, + s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " + + s"pf=${bytesPf.value}") + } + } + } + + // ----- Explain output ----- + + test("StorageFilters shows up in the scan description only when the scan has storage filters") { + // `simpleString` renders every metadata entry verbatim, so an unconditional entry would append + // `StorageFilters: []` to every file-scan explain line and churn the explain golden files. + withTempDir { dir => + val rows = (1L to 20L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows) + val scan = spark.read.parquet(path).select("k", "v").queryExecution.executedPlan + .collect { case s: FileSourceScanExec => s }.head + assert(!scan.simpleString(100).contains("StorageFilters"), + s"a scan with no storage filters must not mention them: ${scan.simpleString(100)}") + + val keyAttr = scan.output.find(_.name == "k").get + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(5L)))) + assert(withSF.simpleString(100).contains("StorageFilters"), + s"a scan with storage filters must mention them: ${withSF.simpleString(100)}") + } + } + + // ----- Missing KEY column end to end (schema evolution) ----- + + // Builds a parquet table whose older file predates a later ADD COLUMN, so that column is missing + // from that file. `addColumnClause` is spliced into the ALTER, e.g. "k BIGINT DEFAULT 7". + private def withEvolvedKeyTable(addColumnClause: String)(body: String => Unit): Unit = { + withTable("evolved") { + spark.sql("CREATE TABLE evolved (id BIGINT) USING parquet") + spark.sql("INSERT INTO evolved VALUES (1), (2), (3)") + spark.sql(s"ALTER TABLE evolved ADD COLUMN $addColumnClause") + spark.sql("INSERT INTO evolved VALUES (4, 40), (5, 50)") + body("evolved") + } + } + + // Attaches `storageFilters` to the scan of `SELECT id, k FROM ` and collects the result. + private def collectWithStorageFilterOnKey( + table: String, + buildFilter: Attribute => Expression): Set[(Long, Option[Long])] = { + val df = spark.sql(s"SELECT id, k FROM $table") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output")) + val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr))) + val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF + // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so + // rows arrive in the scan's own order -- the relation's dataSchema order, not the SELECT's. + // Resolve positions by name rather than assuming they line up. + val idPos = scan.output.indexWhere(_.name == "id") + val kPos = scan.output.indexWhere(_.name == "k") + rowPlan.executeCollect() + .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos)))) + .toSet + } + + test("missing key column with an existence DEFAULT is filtered on the default, not on null") { + // The older file has no `k`, so the reader materializes k = 7 for its rows. The predicate must + // be evaluated against 7, which keeps the file. Evaluating it against null yields null, which + // is not true, so the whole older file would be skipped and its rows lost. + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") { + withEvolvedKeyTable("k BIGINT DEFAULT 7") { table => + val collected = collectWithStorageFilterOnKey( + table, k => GreaterThanOrEqual(k, Literal(5L))) + // k reads as 7 for the old rows (7 >= 5, kept) and as 40/50 for the new ones. + val expected: Set[(Long, Option[Long])] = + Set((1L, Some(7L)), (2L, Some(7L)), (3L, Some(7L)), (4L, Some(40L)), (5L, Some(50L))) + assert(collected == expected, + s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}") + } + } + } + + test("missing key column with an existence DEFAULT that fails the filter skips the older file") { + // Mirror image of the previous test: the default does NOT satisfy the predicate, so the older + // file must be skipped in full while the newer file is still filtered normally. + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") { + withEvolvedKeyTable("k BIGINT DEFAULT 7") { table => + val collected = collectWithStorageFilterOnKey( + table, k => GreaterThanOrEqual(k, Literal(30L))) + val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L))) + assert(collected == expected, + s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}") + } + } + } + + test("missing key column with no DEFAULT reads as null and the predicate decides on null") { + // Without a DEFAULT the column really does read as null, so a null-rejecting predicate skips + // the older file and a null-accepting one keeps it. Both directions are checked so the test + // pins the semantics rather than just one outcome. + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + withEvolvedKeyTable("k BIGINT") { table => + val rejectsNull = collectWithStorageFilterOnKey( + table, k => GreaterThanOrEqual(k, Literal(5L))) + assert(rejectsNull == Set((4L, Some(40L)), (5L, Some(50L))), + s"a null-rejecting predicate should drop the older file; got ${rejectsNull.toSeq.sorted}") + + val acceptsNull = collectWithStorageFilterOnKey( + table, k => Or(IsNull(k), GreaterThanOrEqual(k, Literal(45L)))) + val expected: Set[(Long, Option[Long])] = + Set((1L, None), (2L, None), (3L, None), (5L, Some(50L))) + assert(acceptsNull == expected, + s"a null-accepting predicate should keep the older file; got ${acceptsNull.toSeq.sorted}") + } + } + } + + // ----- Metadata columns and complex non-key columns ----- + + test("_metadata.row_index is correct alongside a spliced key column") { + // The row-index slot is a synthetic non-key slot fed by ParquetRowIndexUtil from the phase-2 + // PageReadStore. It must report absolute row indexes within the file, not positions within the + // filtered batch. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val df = spark.read.parquet(path).select( + col("k"), col("v"), col("_metadata.row_index").as("ri")) + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output")) + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(150L)))) + val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF + val collected = rowPlan.executeCollect().map(r => (r.getLong(0), r.getLong(2))).toSet + // Rows were written in ascending k order in a single file, so row_index == k - 1. + val expected = (150L to 200L).map(k => (k, k - 1)).toSet + assert(collected == expected, + s"row_index must be the absolute index in the file; got " + + s"${collected.toSeq.sorted.take(5)}") + } + } + } + + test("complex non-key column is assembled correctly under splicing") { + // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a + // struct + // column exercises both, which a flat projection never does. + withTempDir { dir => + val outDir = new File(dir, "structs").getAbsolutePath + spark.range(1, 201) + .selectExpr("id AS k", "named_struct('a', CAST(id AS INT), 'b', CONCAT('s_', id)) AS s") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, 256L) + .parquet(outDir) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val df = spark.read.parquet(outDir).select("k", "s") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + val keyAttr = scan.output.find(_.name == "k").getOrElse(fail("no k in scan output")) + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(195L)))) + val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF + val collected = rowPlan.executeCollect() + .map { r => + val s = r.getStruct(1, 2) + (r.getLong(0), s.getInt(0), s.getString(1)) + }.toSet + val expected = (195L to 200L).map(k => (k, k.toInt, s"s_$k")).toSet + assert(collected == expected, s"got ${collected.toSeq.sorted}; expected $expected") + } + } + } + + // ----- Planner gates and the lost-filter invariant ----- + + test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") { + // The whole lost-filter safety argument rests on this gate: if the reader cannot do late + // materialization, the planner must NOT move the bloom out of the post-scan Filter. + withBloomFilterTables { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val (plan, _) = runBloomFilterJoin() + assert(countBloomFiltersInStorageFilters(plan) == 0, + s"no bloom should be extracted when the vectorized reader is off.\nPlan:\n$plan") + assert(countBloomFiltersInPostScanFilters(plan) >= 1, + s"the bloom must remain as a post-scan FilterExec.\nPlan:\n$plan") + } + } + } + + test("a scan with storage filters fails loudly if the vectorized reader is disabled later") { + // preparedStorageFilters deliberately does not re-check the conf, because by then the bloom is + // already gone from the post-scan Filter. So a vectorized-reader conf flipped between planning + // and execution must fail rather than quietly return every row. + withTempDir { dir => + val rows = (1L to 50L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows) + val withSF = withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + scanWithStorageFilter(path, "k", threshold = 25L) + } + withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") { + val e = intercept[Exception] { + executePlanCollect(withSF) + } + val message = Option(e.getCause).map(_.getMessage).getOrElse(e.getMessage) + assert(message != null && message.contains("Cannot honor storage filters"), + s"expected a clear storage-filter failure; got: $message") + } + } + } + + test("a file format without storage-filter support rejects a non-empty storageFilters") { + // The default `FileFormat.buildReaderWithStorageFilters` body must not drop the filters it is + // handed: extraction has already removed them from the post-scan Filter, so a reader that + // ignores them returns rows the filter rejects. Only the planner's + // `getClass == classOf[ParquetFileFormat]` gate keeps this unreachable today, and that gate + // lives in another file. + val storageFilters = + Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L))) + val e = intercept[IllegalArgumentException] { + new NoStorageFilterFileFormat().buildReaderWithStorageFilters( + spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters, + Map.empty, new Configuration()) + } + assert(e.getMessage.contains("does not support storage-filter pushdown"), e.getMessage) + } + + Seq(false, true).foreach { aqe => + test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") { + // AQE is on by default in production, and it is where the bloom subquery is planned by + // PlanAdaptiveSubqueries rather than PlanSubqueries -- the path preparedStorageFilters' + // ScalarSubquery materialization depends on. + withBloomFilterTables { + val baseConf = Map( + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) + def run(pushdown: Boolean): Set[(Long, Long)] = withSQLConf( + (baseConf + + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _* + ) { + runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet + } + val off = run(false) + val on = run(true) + assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off") + assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing") + } + } + } + + test("canonicalization keeps a storage-filter scan distinct from a plain one") { + // storageFilters is in doCanonicalize and in the case class equality, which is what stops + // exchange and subquery reuse from serving one scan's result to the other. Reuse compares + // canonicalized plans, so a scan that filters must not match a plain scan of the same file, and + // two scans carrying the same filter must still match. + withTempDir { dir => + val rows = (1L to 50L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val withSF = scanWithStorageFilter(path, "k", threshold = 25L) + val plain = withSF.copy(storageFilters = Nil) + assert(withSF != plain, "case class equality must take storageFilters into account") + assert(!withSF.sameResult(plain), + s"a filtering scan must not be reusable as a plain one:\n${withSF.canonicalized}\n" + + s"${plain.canonicalized}") + // A second, independently planned scan of the same file with the same filter still + // matches, so reuse is not disabled wholesale. Its key attribute carries a different + // exprId, which is what canonicalization normalizes away. + val sameSF = scanWithStorageFilter(path, "k", threshold = 25L) + assert(withSF.sameResult(sameSF), + s"two scans with the same storage filter must stay reusable:\n" + + s"${withSF.canonicalized}\n${sameSF.canonicalized}") + val otherSF = scanWithStorageFilter(path, "k", threshold = 30L) + assert(!withSF.sameResult(otherSF), "a different threshold is a different result") + } + } + } + + test("whole-stage codegen off: the row-at-a-time path still applies the extracted bloom") { + // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at + // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off + // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced + // batch + // one row at a time. That combination is the only one where the planner's gate is weaker than + // the runtime's, and only the reader-level test covered the row-at-a-time path. + withBloomFilterTables { + val baseConf = Map( + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false", + SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "false") + def run(pushdown: Boolean): (Set[(Long, Long)], Int, Boolean) = withSQLConf( + (baseConf + + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _* + ) { + val (plan, result) = runBloomFilterJoin() + val columnar = plan.collect { case s: FileSourceScanExec => s.supportsColumnar } + (result.map(r => (r.getLong(0), r.getLong(1))).toSet, + countBloomFiltersInStorageFilters(plan), columnar.forall(_ == false)) + } + val (off, _, _) = run(false) + val (on, storageBlooms, noColumnarScan) = run(true) + assert(storageBlooms >= 1, s"the bloom must still be extracted with codegen off; " + + s"got $storageBlooms") + assert(noColumnarScan, "with codegen off no scan should output columnar batches") + assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off") + assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing") + } + } + + test("off-heap column vectors through the planner: spliced values survive the free") { + // Off-heap is where the vector lifecycle actually bites: the previous batch's key vectors are + // freed at the next nextBatch(), so a stale reference reads released native memory rather than + // old bytes. Only the reader-level tests passed useOffHeap = true; this drives it through the + // planner, where the batch also crosses ColumnarToRowExec. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> "true") { + val scan = scanWithStorageFilter(path, "k", threshold = 150L) + val collected = executePlanCollect(scan).toSet + val expected = rows.filter(_._1 >= 150L).toSet + assert(collected == expected, + s"got ${collected.size} rows; expected ${expected.size}. " + + s"first few: ${collected.toSeq.sortBy(_._1).take(3)}") + } + } + } + + // ----- Page-level pushedFilterRanges (a strict subset of the row group) ----- + + test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") { + // Every other fixture writes one page per column per row group, so column-index filtering can + // only ever drop whole row groups and `pushedFilterRanges` is always the entire block. That + // makes the phase 1 alignment -- the r-th row readBatch delivers must pair with + // rowIndexIter.nextLong() -- hold trivially. With a small page size the data filter narrows + // to a page subset, so the two sequences only agree if the pairing is actually correct. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + // `v` is correlated with `k`, so a range predicate on v prunes pages, not whole row groups. + val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0300'") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + assert(scan.simpleString(200).contains("GreaterThanOrEqual(v,"), + s"the data filter must be pushed: ${scan.simpleString(200)}") + val keyAttr = scan.output.find(_.name == "k").get + // Storage filter keeps a band that starts inside the data filter's surviving range. + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L)))) + + val collected = executePlanCollect(withSF).toSet + val expected = rows.filter(r => r._2 >= "v_0300" && r._1 >= 350L).toSet + assert(collected == expected, + s"got ${collected.size} rows; expected ${expected.size}. " + + s"first few: ${collected.toSeq.sortBy(_._1).take(3)}") + } + } + } + + test("page-subset ranges keep the avoided-byte metrics non-negative") { + // The strict-subset branch of compressedBytesForRowRanges (offset index + dictionary page) only + // runs when pushedFilterRanges is narrower than the block, which needs a multi-page row group. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val df = spark.read.parquet(path).select("k", "v").filter("v >= 'v_0100'") + val scan = df.queryExecution.executedPlan + .collect { case s: FileSourceScanExec => s }.head + val keyAttr = scan.output.find(_.name == "k").get + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(390L)))) + val collected = executePlanCollect(withSF).toSet + assert(collected == rows.filter(_._1 >= 390L).toSet, s"got ${collected.size} rows") + + val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) + val bytesPf = + withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING) + assert(bytesRg.value >= 0 && bytesPf.value >= 0, + s"avoided-byte metrics must never go negative; rg=${bytesRg.value} pf=${bytesPf.value}") + } + } + } + + test("column-index filtering off: phase 0 takes the whole row group") { + // Phase 0 asks parquet for row ranges only when column-index filtering is on, because + // ParquetFileReader.getRowRanges checks whether a filter is pushed and NOT whether the user + // enabled the column index. That branch is the escape hatch for a file whose column index is + // wrong -- trusting one here would drop rows for good, since the post-scan Filter no longer + // holds the predicate -- and nothing exercised it. + // + // The row accounting is what tells the two arms apart. Everything is scoped to the rows the + // pushed data filter left, so with the column index on, the rows it prunes at page level never + // reach phase 1 and are never counted. With it off, every row of the block does, so emitted + // plus excluded covers the whole file. One row group with many pages keeps statistics-level + // row-group filtering out of it, which happens either way. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + + def run(columnIndex: Boolean): (Set[(Long, String)], Long) = withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED -> columnIndex.toString) { + val df = spark.read.parquet(path).select("k", "v").filter("k >= 350") + val scan = df.queryExecution.executedPlan + .collect { case s: FileSourceScanExec => s }.head + assert(scan.simpleString(200).contains("GreaterThanOrEqual(k,"), + s"the data filter must be pushed for this test to mean anything: " + + scan.simpleString(200)) + val keyAttr = scan.output.find(_.name == "k").get + val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L)))) + val collected = executePlanCollect(withSF).toSet + val accounted = collected.size + + withSF.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value + + withSF.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value + (collected, accounted) + } + + val expected = rows.filter(_._1 >= 350L).toSet + val (rowsOff, accountedOff) = run(columnIndex = false) + val (rowsOn, accountedOn) = run(columnIndex = true) + assert(rowsOff == expected, + s"with the column index off, got ${rowsOff.size} rows; expected ${expected.size}") + assert(rowsOn == expected, + s"with the column index on, got ${rowsOn.size} rows; expected ${expected.size}") + assert(accountedOff == rows.size, + s"with the column index off every row of the file must be emitted or excluded; " + + s"accounted $accountedOff of ${rows.size}") + assert(accountedOn < rows.size, + s"with the column index on the pruned pages must not reach phase 1; " + + s"accounted $accountedOn of ${rows.size}") + } + } + + Seq(true, false).foreach { pushDataFilter => + test("the byte metrics cost no extra IO " + + s"(pushed data filter narrowing the row group = $pushDataFilter)") { + // The whole point of the byte metrics is to report IO that did not happen, so they must not + // cause any. Two arms of the same read, one with all five metrics wired and one with none, + // over a filesystem that counts every byte a read hands back. `needBytes` is false in the + // second arm, so it skips the walks entirely, and any difference in bytes read is the walks'. + // + // Both range shapes are covered, because the walk answers them from different places. With a + // pushed data filter the column index narrows `pushedFilterRanges` to a page subset and the + // walk reads the offset index, which is free only because column-index filtering built and + // memoized the store first. Without one the range is the whole block and the answer comes + // from the footer's `getTotalSize()`, which is the case where nothing else has built that + // store. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + val schema = StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))) + val storageFilters = + Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))) + val pushedFilters = + if (pushDataFilter) Seq(sources.GreaterThan("v", "v_0100")) else Nil + + def run(metrics: Map[String, SQLMetric]): (Int, Long) = { + val hadoopConf = spark.sessionState.newHadoopConf() + hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl", + classOf[CountingLocalFileSystem].getName) + hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true) + val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters( + spark, schema, new StructType(), schema, pushedFilters, storageFilters, + Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics) + val file = PartitionedFile( + InternalRow.empty, + SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"), + 0, + new File(path).length()) + CountingLocalFileSystem.reset() + val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map { + case batch: ColumnarBatch => batch.numRows() + case _ => 1 + }.sum + (emitted, CountingLocalFileSystem.bytesRead()) + } + + val wired = Map( + FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED -> + SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped"), + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP -> + SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup"), + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP -> + SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup"), + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP -> + SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRowGroup"), + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING -> + SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPageFiltering")) + + val (emittedOff, bytesOff) = run(Map.empty) + val (emittedOn, bytesOn) = run(wired) + + assert(emittedOn == emittedOff && emittedOn == 51, + s"both arms must emit keys 350..400; got on=$emittedOn off=$emittedOff") + assert(bytesOn == bytesOff, + s"wiring the byte metrics must not read a single extra byte; " + + s"with metrics $bytesOn, without $bytesOff") + assert(bytesOff > 0, "the counting filesystem must have seen the read at all") + + // The walks really ran, and on the shape each arm is meant to exercise. + val accounted = emittedOn + + wired(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value + + wired(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value + if (pushDataFilter) { + assert(accounted < rows.size, + s"the pushed filter must narrow the ranges below the block, so the offset-index " + + s"branch is the one measured; accounted $accounted of ${rows.size}") + } else { + assert(accounted == rows.size, + s"with no pushed filter every row reaches phase 1, so the footer branch is the one " + + s"measured; accounted $accounted of ${rows.size}") + } + assert(wired(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP).value > 0 || + wired(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING).value > 0, + "at least one byte metric must be non-zero, otherwise the walk answered nothing") + } + } + } + + // ----- Metric arithmetic ----- + + test("row metrics account for every row of the file") { + // Ties the three count metrics to the file: whatever is not emitted must have been avoided + // either by a whole-row-group skip or by page filtering. A sign flip or a mis-scoped schema in + // the accounting shows up here, which a `>= 0` assertion cannot catch. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val scan = scanWithStorageFilter(path, "k", threshold = 150L) + val emitted = executePlanCollect(scan).length + val avoidedRg = scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) + val avoidedPf = + scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP) + assert(emitted == 51, s"expected keys 150..200; got $emitted") + assert(emitted + avoidedRg.value + avoidedPf.value == rows.size, + s"emitted ($emitted) + avoided by row group (${avoidedRg.value}) + avoided by page " + + s"filtering (${avoidedPf.value}) should equal ${rows.size}") + } + } + } + + test("all-key projection: byte metrics are zero and no offset index work is needed") { + // With every projected column a key, phase 2 never runs, so baseline == phase1 and both byte + // metrics are 0 by construction. This is the shape the design notes call the biggest win, so it + // must not be the shape that pays for metrics. + withTempDir { dir => + val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 200L) + val bytesRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") + val bytesPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") + val rowsRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") + val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(190L)) + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested, StorageFilterMetrics( + rowsExcludedByRowGroup = rowsRg, + bytesAvoidedByRowGroup = bytesRg, + bytesAvoidedByPageFiltering = bytesPf)) + val (result, reader) = + readAllWith(path, Seq("k"), filter, (b, i) => b.column(0).getLong(i)) + try { + assert(result == (190L to 200L), s"expected keys 190..200; got $result") + assert(bytesRg.value == 0 && bytesPf.value == 0, + s"an all-key projection has no non-key bytes to avoid; got rg=${bytesRg.value} " + + s"pf=${bytesPf.value}") + assert(rowsRg.value > 0, "row groups should still be skipped, and counted in rows") + } finally { + reader.close() + } + } + } + + // ----- Projection order and batch boundaries ----- + + test("non-key column before the key column: emit maps queues to the right batch slots") { + // The emit loop walks batch slots in ascending order and pulls survivor queues in order, so it + // relies on keyColumnIndices being sorted. Every other test puts the keys in the leading slots, + // where an off-by-one in that pairing is invisible. + withTempDir { dir => + val outDir = new File(dir, "vk").getAbsolutePath + spark.range(1, 201).selectExpr("CONCAT('v_', id) AS v", "id AS k", "id * 10 AS w") + .repartition(1) + .write + .option(ParquetOutputFormat.BLOCK_SIZE, 256L) + .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .parquet(outDir) + val path = new File(outDir).listFiles((_, n) => n.endsWith(".parquet"))(0).getAbsolutePath + + // Key is `k`, at slot 1 of the (v, k, w) projection. + val bound = GreaterThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(195L)) + val requested = StructType(Seq( + StructField("v", StringType, nullable = true), + StructField("k", LongType, nullable = true), + StructField("w", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + assert(filter.keyColumnIndices.toSeq == Seq(1), "the key must be recognized at slot 1") + + val (result, reader) = readAllWith(path, Seq("v", "k", "w"), filter, + (b, i) => (b.column(0).getUTF8String(i).toString, b.column(1).getLong(i), + b.column(2).getLong(i))) + try { + val expected = (195L to 200L).map(k => (s"v_$k", k, k * 10)) + assert(result == expected, s"got $result; expected $expected") + } finally { + reader.close() + } + } + } + + test("survivor count is an exact multiple of capacity: no partial trailing accumulator") { + // finalizePartialAccumulators' early return only runs when the last accumulator is exactly + // full. 64 survivors at capacity 16 hits it; the multi-batch tests use 100, which does not. + withTempDir { dir => + val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 64L, rowGroupSize = 64 * 1024L) + val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(1L)) + val requested = StructType(Seq(StructField("k", LongType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + val (result, reader) = readAllWith( + path, Seq("k"), filter, (b, i) => b.column(0).getLong(i), capacity = 16) + try { + assert(result == (1L to 64L), s"expected all 64 keys in order; got ${result.size}") + } finally { + reader.close() + } + } + } + + test("early termination: the reader stops without draining the file") { + // executeTake on a bare ColumnarToRowExec goes through ColumnarToRowEvaluatorFactory, not + // through the generated code, so nothing closes the batch from outside here. What this covers + // is abandoning the reader mid-file: the survivor queues still hold vectors, and + // RecordReaderIterator closes the reader on task completion. The external close is the test + // below. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + val scan = scanWithStorageFilter(path, "k", threshold = 50L) + val limited = ColumnarToRowExec(scan).executeTake(5) + assert(limited.length == 5, s"expected 5 rows from the limit; got ${limited.length}") + assert(limited.forall(_.getLong(0) >= 50L), + s"every row must satisfy the storage filter; got ${limited.map(_.getLong(0)).toSeq}") + } + } + } + + test("a limit under whole-stage codegen closes the spliced batch from outside") { + // `batch.close()` is emitted by ColumnarToRowExec.doProduce alone, so it only runs under + // WholeStageCodegenExec, and only when the row loop exits with a batch still in hand. That exit + // is the limit check, which needs a limit inside the same codegen stage. So the plan is built + // with LocalLimitExec and handed to CollapseCodegenStages, and the generated source is asserted + // to contain the close -- without that, this test would pass for the wrong reason. + // + // What it exercises: the spliced batch's columns are closed from outside while the reader is + // still open, and the reader's own close() then runs over the same vectors. + withTempDir { dir => + val rows = (1L to 200L).map(i => (i, s"v_$i")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true") { + val scan = scanWithStorageFilter(path, "k", threshold = 50L) + val planned = + CollapseCodegenStages().apply(LocalLimitExec(5, ColumnarToRowExec(scan))) + val stage = planned match { + case w: WholeStageCodegenExec => w + case other => fail(s"expected a whole-stage codegen plan, got $other") + } + val source = stage.doCodeGen()._2.body + assert(source.contains(".close();"), + s"the generated code must close the batch on the limit exit; source:\n$source") + + val limited = stage.executeCollect() + assert(limited.length == 5, s"expected 5 rows from the limit; got ${limited.length}") + assert(limited.forall(_.getLong(0) >= 50L), + s"every row must satisfy the storage filter; got ${limited.map(_.getLong(0)).toSeq}") + } + } + } + + // ----- Partially-missing key columns, end to end through the reader ----- + + test("one of two key columns missing from a file: splicing runs with the rewritten predicate") { + // The most intricate branch of initializeLateMaterialization: splicing engages with a predicate + // that has one Literal substituted and one BoundReference renumbered, the missing key's field + // lands in nonKeyRequestedSchema, and its output slot is filled by ParquetColumnVector. Only + // the all-missing case was covered end to end before. + // + // The SELECT order (a, b, c) also differs from the table's (a, c, b), so this covers a + // projection whose order does not match the relation's dataSchema. + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") { + withTable("partial") { + spark.sql("CREATE TABLE partial (a BIGINT, c STRING) USING parquet") + spark.sql("INSERT INTO partial VALUES (1, 'x'), (2, 'y'), (3, 'z')") + spark.sql("ALTER TABLE partial ADD COLUMN b BIGINT DEFAULT 7") + spark.sql("INSERT INTO partial VALUES (4, 'p', 40), (5, 'q', 50)") + + val df = spark.sql("SELECT a, b, c FROM partial") + val plan = df.queryExecution.executedPlan + val scan = plan.collect { case s: FileSourceScanExec => s }.headOption + .getOrElse(fail(s"No FileSourceScanExec in plan: $plan")) + val a = scan.output.find(_.name == "a").getOrElse(fail("no a")) + val b = scan.output.find(_.name == "b").getOrElse(fail("no b")) + // Two key columns. In the older file `b` is missing and reads as its default 7, so the + // predicate must be evaluated with 7 substituted for it -- and `a >= 2` still filters. + val withSF = scan.copy(storageFilters = Seq( + GreaterThanOrEqual(a, Literal(2L)), GreaterThanOrEqual(b, Literal(5L)))) + + val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF + // The scan emits its own order (a, c, b here), not the SELECT's (a, b, c), because + // executing it directly skips the reordering Project. Resolve positions by name. + val aPos = withSF.output.indexWhere(_.name == "a") + val bPos = withSF.output.indexWhere(_.name == "b") + val cPos = withSF.output.indexWhere(_.name == "c") + val collected = rowPlan.executeCollect() + .map(r => (r.getLong(aPos), r.getString(cPos), r.getLong(bPos))).toSet + // Older file: a in {2,3} pass a>=2, and b=7 passes b>=5. Newer file: 40 and 50 both pass. + val expected = Set((2L, "y", 7L), (3L, "z", 7L), (4L, "p", 40L), (5L, "q", 50L)) + // Compare against the plain read too, so a failure here is unambiguously the storage-filter + // path rather than a wrong expectation. df.collect() goes through the Project, so it is in + // the SELECT order (a, b, c). + val baseline = df.collect().map(r => (r.getLong(0), r.getString(2), r.getLong(1))).toSet + assert(baseline == expected + ((1L, "x", 7L)), + s"the plain read is already wrong, so the expectation is: ${baseline.toSeq.sorted}") + assert(collected == expected, + s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}") + } + } + } +} + +/** + * A [[FileFormat]] that does not override `buildReaderWithStorageFilters`, so it exercises the + * default body's rejection of storage filters it cannot honor. + */ +private class NoStorageFilterFileFormat extends FileFormat { + override def inferSchema( + sparkSession: SparkSession, + options: Map[String, String], + files: Seq[FileStatus]): Option[StructType] = None + + override def prepareWrite( + sparkSession: SparkSession, + job: Job, + options: Map[String, String], + dataSchema: StructType): OutputWriterFactory = + throw new UnsupportedOperationException("write is not supported by this test format") +} + +/** + * A local filesystem under its own scheme that counts the bytes every read hands back, so a test + * can compare the IO of two runs. The wrapper does not implement `ByteBufferReadable`, so reads go + * through the byte-array path, which is fine as long as both runs use this same filesystem. + */ +class CountingLocalFileSystem extends RawLocalFileSystem { + override def getUri: URI = URI.create(s"${CountingLocalFileSystem.scheme}:///") + + override def open(f: Path, bufferSize: Int): FSDataInputStream = + new FSDataInputStream(new CountingLocalFileSystem.CountingStream(super.open(f, bufferSize))) +} + +object CountingLocalFileSystem { + val scheme = "countingfile" + + private val counter = new AtomicLong(0L) + + def reset(): Unit = counter.set(0L) + + def bytesRead(): Long = counter.get() + + private class CountingStream(in: FSDataInputStream) extends FSInputStream { + override def seek(pos: Long): Unit = in.seek(pos) + + override def getPos: Long = in.getPos + + override def seekToNewSource(targetPos: Long): Boolean = in.seekToNewSource(targetPos) + + override def read(): Int = { + val b = in.read() + if (b >= 0) counter.incrementAndGet() + b + } + + override def read(buf: Array[Byte], off: Int, len: Int): Int = { + val n = in.read(buf, off, len) + if (n > 0) counter.addAndGet(n) + n + } + + override def close(): Unit = in.close() + } +} From b9981cda93b0d28224168f7449fbcc59176e327d Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Wed, 23 Sep 2026 12:22:03 +0200 Subject: [PATCH 2/6] Review fixes Addresses dongjoon-hyun's review of 2026-09-22. What changed, comment by comment: - Resource lifecycle. `pendingCloseKeyVectors` is gone: a survivor vector now stays owned by its queue while the batch is built on it, and the next batch closes it, so nothing can be left unreachable by `close()`. The phase-1 scratch vectors are closed through their array rather than through the batch that is assigned after the allocation loop, so a failed allocation cannot leak either. - `ParquetFileFormat`'s two public reader builders both route to a private `buildParquetReader`, so a subclass that overrides one and delegates to the other cannot recurse. The trait's scaladoc says so. - `resultBatch()` hands out one batch for the whole read again, storage filter or not. The emit path rewrites the batch's key slots in place instead of building a new `ColumnarBatch` per batch, which holds the method's documented contract for every caller and drops two allocations per batch. - The two unused `protected` fields are gone from `SpecificParquetRecordReaderBase`, and the footer is a local again. - Per-row-group work removed from the byte metrics: the three leaf-column lists are resolved once per file, the block's path-to-chunk map is built once per row group, and the row counts the caller already has are passed in instead of recomputed. - The dead `VarcharType` and `CharType` branches are gone from `copierFor`. - The test helpers drain inside `Utils.tryInitializeResource`, so a failure in the read loop closes the reader instead of leaking it. - The offset index is no longer checked up front. Phase 2's read is wrapped and parquet's own `MissingOffsetIndexException` rethrown with guidance. That is narrower, since a row group the filter keeps whole never needs the index, and more complete, since an IO error while reading an index surfaces the same way. It also made the change a net deletion. - New `UnsupportedFileReadException`, excluded from `DataSourceUtils.shouldIgnoreCorruptFileException` and thrown by every loud failure in this path. Without it `ignoreCorruptFiles` read those failures as a corrupt file and silently dropped the rest of a healthy file's rows. - The survivor buffer is bounded per row group by `spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes` (internal, 64MB). Phase 1 counts the bytes it has buffered and gives that row group up once the count passes the cap, releasing what it holds; phase 2 then reads every projected column of the surviving rows, which buffers nothing and costs one extra read of the key columns. The count is examined once per accumulator, so it costs one comparison per batch worth of survivors rather than one per row. - `FileFormat.supportsStorageFilter(expr)`, defaulting to false, replaces the planner's exact-class check and its call into `ParquetStorageFilter`. `FileSourceStrategy` no longer imports the parquet package, and the reader's key-type list lives next to the copier that defines it. - Extraction is dropped when every projected data column is a key column of the filter. Such a scan reads the same columns for the same rows either way, since the reader has to read a key column to evaluate the filter on it, so pushing could only add the cost of evaluating the predicate outside the generated code. TPCDS q37 is that shape, and measured 3199 ms with the feature off against 3949 ms with it on. - `bytesAvoidedByPageFiltering` charges the second read of the key columns to the row group that gave splicing up. It credited the full non-key saving before and counted the extra read nowhere, so it reported a saving for a row group that transferred more bytes than a plain read would have. - `isSupportedStorageFilter` type-checks every reference of the conjunct, not only the ones on the bloom's value side, which is what `create` binds and re-checks. - Both confs declare a binding policy, `NOT_APPLICABLE`: neither can change how a view, UDF or procedure body resolves. - `.version("5.0.0")` stays. The feature needs the Parquet 1.18 upgrade, which landed on master only. - Comments and scaladoc across the change are trimmed to what the current code needs. Three comments are answered on the PR rather than in code. Chunk-at-a-time phase-1 evaluation is not worth it: parquet reads a row group's whole requested chunks before it hands back a page store, so the IO of a `LIMIT` query is the same either way and what the first batch waits for is one row group's key decode, while chunking would multiply the read calls and refetch pages that straddle a chunk boundary. The per-row `byte[]` in the string copier wants a vector-to-vector `appendBytes` on `WritableColumnVector`, which belongs in its own change. Reusing `RowToColumnConverter` for the copiers would put the row abstraction back in the hot loop, which is the opposite direction from that one. Verified: `ParquetStorageFilterSuite` 95 tests, plus `ParquetIOSuite`, `FileSourceStrategySuite`, `ParquetV1FilterSuite`, `ParquetV2FilterSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `DataFrameJoinSuite` and `SubquerySuite`, 664 in all; `dev/lint-scala`; `dev/lint-java`. --- .../apache/spark/sql/internal/SQLConf.scala | 28 +- .../SpecificParquetRecordReaderBase.java | 23 +- .../VectorizedParquetRecordReader.java | 732 ++++++++++-------- .../sql/execution/DataSourceScanExec.scala | 36 +- .../datasources/DataSourceUtils.scala | 8 + .../execution/datasources/FileFormat.scala | 24 +- .../datasources/FileSourceStrategy.scala | 83 +- .../UnsupportedFileReadException.scala | 35 + .../parquet/ParquetFileFormat.scala | 48 +- .../parquet/ParquetStorageFilter.scala | 125 ++- .../parquet/ParquetStorageFilterSuite.scala | 281 +++++-- 11 files changed, 871 insertions(+), 552 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 3c2f18eeb5d9c..3d6bf4b198992 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1894,12 +1894,33 @@ object SQLConf { "false, no storage filter is attached to a scan in the first place and the filter is " + "applied as an ordinary post-scan filter instead. Note that the surviving key values of " + "a whole row group are buffered before the first batch of that row group is produced, " + - "so a task holds up to one extra copy of the key columns for one row group.") + "so a task holds up to one extra copy of the key columns for one row group. Note also " + + "that reading only the surviving rows of a row group needs a Parquet offset index. A " + + "scan of a file written without a page index fails as soon as its filter rejects part " + + "of a row group, so set this to false to read such a file.") .version("5.0.0") - .withBindingPolicy(ConfigBindingPolicy.SESSION) + .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) .booleanConf .createWithDefault(false) + val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES = + buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes") + .internal() + .doc("Largest key-column buffer, in bytes, that the vectorized Parquet reader will hold to " + + "splice surviving key values into its output batches. Splicing buffers one key value per " + + "surviving row of a row group, so the reader counts what it has buffered and gives that " + + "row group up once the count passes this, reading every projected column of the " + + "surviving rows in one go instead, which costs one extra read of the key columns. What " + + "is counted is the buffered values and their per-row overhead, not the backing arrays, " + + "which a column vector may grow beyond that. The count is examined whenever a batch " + + "worth of survivors per key column has been buffered, so a row group whose survivors fit " + + "in a single batch is never given up: it holds no more than the plain read path does.") + .version("5.0.0") + .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) + .bytesConf(ByteUnit.BYTE) + .checkValue(_ > 0, "must be positive") + .createWithDefaultString("64MB") + val PARQUET_FILTER_PUSHDOWN_DATE_ENABLED = buildConf("spark.sql.parquet.filterPushdown.date") .doc("If true, enables Parquet filter push-down optimization for Date. " + s"This configuration only has an effect when '${PARQUET_FILTER_PUSHDOWN_ENABLED.key}' is " + @@ -9124,6 +9145,9 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def parquetStorageFilterPushdownEnabled: Boolean = getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED) + def parquetStorageFilterPushdownMaxSplicedRowGroupBytes: Long = + getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES) + def parquetFilterPushDownDate: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_DATE_ENABLED) def parquetFilterPushDownTimestamp: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_TIMESTAMP_ENABLED) diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java index 11fe908eaa10b..43b6b6bd81107 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java @@ -87,13 +87,6 @@ public abstract class SpecificParquetRecordReaderBase extends RecordReader columns) throws IOException .builder(configuration, file) .withRange(0, length) .build(); - this.inputFile = HadoopInputFile.fromPath(file, configuration); - ParquetFileReader fileReader = ParquetFileReader.open(this.inputFile, options); - this.fileFooter = fileReader.getFooter(); + ParquetFileReader fileReader = ParquetFileReader.open( + HadoopInputFile.fromPath(file, configuration), options); this.reader = new ParquetRowGroupReaderImpl(fileReader); - this.fileSchema = fileFooter.getFileMetaData().getSchema(); + this.fileSchema = fileReader.getFooter().getFileMetaData().getSchema(); if (columns == null) { this.requestedSchema = fileSchema; diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index 2252dd93b5564..e178f3c12ea29 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -57,6 +57,7 @@ import org.apache.spark.memory.MemoryMode; import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns; import org.apache.spark.sql.catalyst.InternalRow; +import org.apache.spark.sql.execution.datasources.UnsupportedFileReadException; import org.apache.spark.sql.execution.metric.SQLMetric; import org.apache.spark.sql.execution.vectorized.ColumnVectorUtils; import org.apache.spark.sql.execution.vectorized.ConstantColumnVector; @@ -181,14 +182,37 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa * *

One {@link ParquetFileReader} ({@link #lateMatReader}, the base class's reader exposed via * {@link ParquetRowGroupReader#getUnderlyingReader()}) drives all three phases. Its requested - * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: full schema for - * phase 0 ({@code getRowRanges}), key-only for phase 1, non-key only for phase 2 (or skipped - * entirely when the projected schema is all keys, indicated by {@link #nonKeyRequestedSchema} - * being null). + * schema is mutated per phase via {@link ParquetFileReader#setRequestedSchema}: all projected + * columns for phase 0 ({@code getRowRanges}), the key columns for phase 1, the non-key columns + * for phase 2 (or skipped entirely when the projection is all keys, which is what + * {@link #nonKeyColumns} being null means). + * + *

The three sets are held as leaf-column lists rather than as {@link MessageType}s because + * that is what both the reader and the byte metrics consume, and because + * {@link MessageType#getColumns()} rebuilds the list on every call. */ private ParquetFileReader lateMatReader; - private MessageType keyOnlyRequestedSchema; - private MessageType nonKeyRequestedSchema; + private List requestedColumns; + private List keyOnlyColumns; + private List nonKeyColumns; + /** + * What an accumulator holds per row for each key column, not counting the value bytes of a + * variable-length type. Together with those value bytes it is what the per-row-group buffer is + * measured against. See + * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}. + */ + private int keyFixedBytesPerRow; + /** Which key columns hold their values out of line, so a length has to be measured per row. */ + private boolean[] keyVariableLength; + /** Key-value bytes buffered for the row group currently loading. */ + private long splicedBytes; + /** + * Whether the row group currently loaded is spliced. Starts true for every row group and can turn + * false in phase 1, once the survivors buffered pass the cap; false means phase 2 read every + * projected column, key columns included, so the emit path takes them straight from the + * persistent batch. + */ + private boolean spliceCurrentRowGroup; private int nextBlockIndex; private int totalBlockCount; private ColumnDescriptor[] keyDescriptors; @@ -203,25 +227,27 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa private boolean useColumnIndexFilter = true; /** - * Splicing state. {@link #isKeyTopLevel} marks top-level requested-schema slots that are sourced - * from the per-key-column queues instead of phase-2 reads. - * {@link #keyVectorQueues} holds one queue per (present) key column of capacity-sized survivor - * vectors filled during phase 1; {@link #currentKeyAccumulators} are the currently-filling - * vectors not yet pushed to the queue. - * {@link #pendingCloseKeyVectors} stashes the previous emit's dequeued vectors so they can be - * closed at the start of the next emit, releasing survivor memory incrementally as we emit. - * {@link #persistentBatchColumns} captures the original {@link #initBatch} vector array so it can - * be closed separately from the per-emit splicing batch (which shares those vectors and would - * otherwise double-close them). + * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit + * path splices them back into the output batch, so phase 2 never reads the key columns. + * + *

The alternative is for phase 2 to read the key columns again under the surviving row + * ranges, which needs none of this state but pays a second read of them. That second read is + * the whole cost of the scan when the projection is key columns alone, the shape a pushed join + * filter most often produces: splicing skips phase 2 there, while a re-read has nothing left to + * skip. * - *

Phase 2 is skipped entirely when the projected schema is all key columns - * ({@link #nonKeyRequestedSchema} is null); emit reconstructs each batch purely from the key - * queues. + *

{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather + * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of + * capacity-sized survivor vectors, and {@link #currentKeyAccumulators} the vectors still filling. + * A queue keeps owning its head while the batch is built on it, until the next emit closes it, so + * survivor memory drains as the row group is emitted. {@link #persistentBatchColumns} is + * {@link #initBatch}'s vector array and {@link #spliceBatchColumns} the array the emitted batch + * is built over, which is why the two are closed separately. * - *

Memory: phase 1 evaluates the whole row group before the first batch of that row group is - * emitted, so the queues hold every surviving key value for one row group at once -- up to one - * extra copy of the key columns per row group, versus one capacity-sized vector on the plain read - * path. {@link #pendingCloseKeyVectors} then releases them batch by batch as emit progresses. + *

Phase 1 evaluates a whole row group before its first batch is emitted, so the queues hold + * every surviving key value of one row group at once. That is bounded per row group by + * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes}, past which the row + * group is read the plain way and nothing is buffered. */ private boolean[] isKeyTopLevel; private java.util.ArrayDeque[] keyVectorQueues; @@ -230,8 +256,10 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa private int currentKeyAccumulatorRowCount; /** Per-key-column copier picked once at init time; called per surviving row in the hot loop. */ private ValueCopier[] keyCopiers; - private WritableColumnVector[] pendingCloseKeyVectors; + /** Whether each queue's head is the vector the current batch is built on. */ + private boolean keyVectorsPublished; private ColumnVector[] persistentBatchColumns; + private ColumnVector[] spliceBatchColumns; public VectorizedParquetRecordReader( ZoneId convertTz, @@ -313,10 +341,10 @@ public void close() throws IOException { // the rest. try { if (isKeyTopLevel != null) { - // Splicing: the per-emit columnarBatch is a transient view whose slots alias - // persistentBatchColumns (non-key + partition) and pendingCloseKeyVectors (dequeued key - // slots). We never call columnarBatch.close() because that would re-close those shared - // vectors after the direct closes below, freeing the same buffer twice. + // Splicing: the emitted batch is a view whose slots alias persistentBatchColumns (non-key + // and partition) and the head of each survivor queue (key slots). We never call + // columnarBatch.close() because that would re-close those shared vectors after the direct + // closes below, freeing the same buffer twice. try { if (persistentBatchColumns != null) { for (ColumnVector v : persistentBatchColumns) { @@ -324,6 +352,7 @@ public void close() throws IOException { } persistentBatchColumns = null; } + spliceBatchColumns = null; columnarBatch = null; } finally { closeSplicingState(); @@ -335,11 +364,11 @@ public void close() throws IOException { } } finally { try { - if (keyScratchBatch != null) { - keyScratchBatch.close(); - keyScratchBatch = null; - keyScratchVectors = null; - } + // Through the array, not through `keyScratchBatch`: the batch is assigned only after the + // allocation loop finishes, so a partial failure leaves vectors only the array can reach. + closeAll(keyScratchVectors); + keyScratchVectors = null; + keyScratchBatch = null; } finally { // lateMatReader aliases the base-class reader; super.close() owns it. lateMatReader = null; @@ -403,21 +432,25 @@ private void initBatch( constantColumnLength = partitionColumns.fields().length; } - Set keySlotsToSkip = null; - if (isKeyTopLevel != null) { - int[] keyIndices = storageFilter.keyColumnIndices(); - keySlotsToSkip = new HashSet<>(keyIndices.length); - for (int idx : keyIndices) keySlotsToSkip.add(idx); - } + // Every slot gets a vector, key columns included. While splicing, a key slot's vector is unused + // -- the emitted batch takes that slot from the survivor queues -- but a row group read the + // plain way past the buffer cap reads into it, and one capacity-sized vector per key column is + // cheap next to the buffer the cap is there to bound. ColumnVector[] vectors = allocateColumns( - capacity, batchSchema, memMode == MemoryMode.OFF_HEAP, constantColumnLength, keySlotsToSkip); + capacity, batchSchema, memMode == MemoryMode.OFF_HEAP, constantColumnLength); columnarBatch = new ColumnarBatch(vectors); persistentBatchColumns = vectors; + if (isKeyTopLevel != null) { + // Splicing hands out one batch for the whole read, over its own array, whose key slots the + // emit path rewrites in place. `ColumnarBatch` holds the array by reference, its staging row + // included, so rewriting a slot is what publishes it. + spliceBatchColumns = vectors.clone(); + columnarBatch = new ColumnarBatch(spliceBatchColumns); + } columnVectors = new ParquetColumnVector[sparkSchema.fields().length]; for (int i = 0; i < columnVectors.length; i++) { - if (vectors[i] == null) continue; // splicing key slot; ParquetColumnVector unused Object defaultValue = null; if (sparkRequestedSchema != null) { defaultValue = ResolveDefaultColumns.existenceDefaultValues(sparkRequestedSchema)[i]; @@ -500,8 +533,11 @@ static DataType truncateType(DataType readType, DataType requestedType) { /** * Returns the ColumnarBatch object that will be used for all rows returned by this reader. - * This object is reused. Calling this enables the vectorized reader. This should be called - * before any calls to nextKeyValue/nextBatch. + * Calling this enables the vectorized reader. This should be called before any calls to + * nextKeyValue/nextBatch. + * + *

The object is reused, a storage filter included: the reader then rewrites the batch's key + * slots in place with the survivor vectors it spliced for that batch. */ public ColumnarBatch resultBatch() { if (columnarBatch == null) initBatch(); @@ -530,7 +566,29 @@ public boolean nextBatch() throws IOException { if (hitEndOfData) return false; int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); - for (ParquetColumnVector cv : columnVectors) { + readPersistentColumns(num, /* skipKeySlots= */ false); + // If needed, compute row indexes within a file. + if (rowIndexGenerator != null) { + rowIndexGenerator.populateRowIndex(columnVectors, num); + } + finishBatch(num); + return true; + } + + /** + * Reads {@code num} rows into the persistent batch slots and assembles them. All three emit paths + * share this: the plain read, a spliced row group (which skips the key slots, since the emitted + * batch takes those from the survivor queues), and a row group read the plain way past the buffer + * cap (which reads every slot). + * + *

{@code skipKeySlots} is not the same as "the slot has no vector": every slot has one, and a + * key slot's phase-2 column reader is deliberately left unset while splicing, so driving it would + * use whatever a previous row group left behind. + */ + private void readPersistentColumns(int num, boolean skipKeySlots) throws IOException { + for (int i = 0; i < columnVectors.length; i++) { + if (skipKeySlots && isKeyTopLevel[i]) continue; + ParquetColumnVector cv = columnVectors[i]; for (ParquetColumnVector leafCv : cv.getLeaves()) { VectorizedColumnReader columnReader = leafCv.getColumnReader(); if (columnReader != null) { @@ -540,42 +598,30 @@ public boolean nextBatch() throws IOException { } cv.assemble(); } - // If needed, compute row indexes within a file. - if (rowIndexGenerator != null) { - rowIndexGenerator.populateRowIndex(columnVectors, num); - } + } - rowsReturned += num; + /** Publishes {@code num} rows as the current batch. */ + private void finishBatch(int num) { columnarBatch.setNumRows(num); + rowsReturned += num; numBatched = num; batchIdx = 0; - return true; } /** - * Splicing emit path. Closes the previous emit's dequeued key vectors (releasing survivor memory - * incrementally), advances to the next row group if needed via {@link #checkEndOfRowGroup()}, - * dequeues one survivor key vector per key column, drives non-key column readers for {@code num} - * rows, and assembles a fresh {@link ColumnarBatch} interleaving key (dequeued) and non-key - * (persistent value-vector) slots in the original projection order. - * The per-emit batch is a transient view over vectors owned elsewhere; see {@link #close()} and - * {@link #closeSplicingState()}. + * Splicing emit path. Publishes one survivor key vector per key column into the batch, reads the + * non-key slots for {@code num} rows, and hands out the same {@link ColumnarBatch} every time, + * its key slots rewritten in place. The batch is a view over vectors owned elsewhere; see + * {@link #close()} and {@link #closeSplicingState()}. */ private boolean nextBatchSplicing() throws IOException { - if (pendingCloseKeyVectors != null) { - for (WritableColumnVector v : pendingCloseKeyVectors) { - if (v != null) v.close(); - } - pendingCloseKeyVectors = null; - } - for (int i = 0; i < columnVectors.length; i++) { - if (isKeyTopLevel[i]) continue; - columnVectors[i].reset(); + releasePublishedKeyVectors(); + for (ParquetColumnVector cv : columnVectors) { + cv.reset(); } - // Match the eager path and zero the outgoing batch before the terminal checks below. Without - // this, a terminal call leaves `columnarBatch` pointing at the previous emit whose key slots - // were just closed above -- with off-heap vectors those buffers are already freed, so a - // consumer that read the batch after nextBatch() returned false would see freed memory. + // Zero the outgoing batch before the terminal checks below, as the plain path does. Otherwise a + // terminal call leaves the batch pointing at key vectors that were just closed -- off-heap, + // that is freed memory. if (columnarBatch != null) columnarBatch.setNumRows(0); if (hitEndOfData) return false; if (rowsReturned >= totalRowCount) return false; @@ -584,53 +630,67 @@ private boolean nextBatchSplicing() throws IOException { int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); - WritableColumnVector[] dequeued = new WritableColumnVector[keyVectorQueues.length]; - for (int i = 0; i < keyVectorQueues.length; i++) { - dequeued[i] = keyVectorQueues[i].removeFirst(); + if (!spliceCurrentRowGroup) { + // This row group was read the plain way: phase 2 took every projected column, so every slot + // of the batch comes from the persistent one. + readPersistentColumns(num, /* skipKeySlots= */ false); + if (rowIndexGenerator != null) { + rowIndexGenerator.populateRowIndex(columnVectors, num); + } + System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length); + finishBatch(num); + return true; } - for (int i = 0; i < columnVectors.length; i++) { - if (isKeyTopLevel[i]) continue; - ParquetColumnVector cv = columnVectors[i]; - for (ParquetColumnVector leafCv : cv.getLeaves()) { - VectorizedColumnReader columnReader = leafCv.getColumnReader(); - if (columnReader != null) { - columnReader.readBatch(num, leafCv.getValueVector(), - leafCv.getRepetitionLevelVector(), leafCv.getDefinitionLevelVector()); - } + for (int i = 0; i < keyVectorQueues.length; i++) { + if (keyVectorQueues[i].isEmpty()) { + // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop + // is driven by that same count. Named rather than left to NoSuchElementException because + // this one must not be swallowed: under ignoreCorruptFiles that would silently drop the + // rest of a healthy file's rows. + throw new UnsupportedFileReadException(String.format( + "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to " + + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num)); + } + } + // The queues keep owning these until the next emit releases them, so a phase-2 read below that + // throws leaves them reachable for `close()`. + keyVectorsPublished = true; + // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list + // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts + // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each + // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap + // key columns in the output batch. + int keyIdx = 0; + for (int i = 0; i < spliceBatchColumns.length; i++) { + if (i < isKeyTopLevel.length && isKeyTopLevel[i]) { + spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst(); + } else { + spliceBatchColumns[i] = persistentBatchColumns[i]; } - cv.assemble(); } + + readPersistentColumns(num, /* skipKeySlots= */ true); if (rowIndexGenerator != null) { // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a // synthetic metadata column never referenced by a storage filter, so its slot is a non-key // slot with a persistent ParquetColumnVector. rowIndexGenerator.populateRowIndex(columnVectors, num); } + finishBatch(num); + return true; + } - ColumnVector[] cols = new ColumnVector[persistentBatchColumns.length]; - // This walks batch slots in ascending order while `keyIdx` walks the survivor queues in - // key-row-position order, so it pairs the k-th smallest key slot with key-row position k. - // That is only the identity because `ParquetStorageFilter.create` sorts `keyColumnIndices` - // ascending -- see the comment there. `isKeyTopLevel` marks which slots are keys but not their - // position in that list, so this loop cannot reconstruct the pairing on its own: if the list - // ever stops being sorted, key columns silently swap places in the output batch. - int keyIdx = 0; - for (int i = 0; i < persistentBatchColumns.length; i++) { - if (i < isKeyTopLevel.length && isKeyTopLevel[i]) { - cols[i] = dequeued[keyIdx++]; - } else { - cols[i] = persistentBatchColumns[i]; - } + /** + * Closes the key vectors the previous batch was built on. The queues own them until here, so + * survivor memory drains as a row group is emitted rather than all at its end. + */ + private void releasePublishedKeyVectors() { + if (!keyVectorsPublished) return; + keyVectorsPublished = false; + for (java.util.ArrayDeque queue : keyVectorQueues) { + queue.removeFirst().close(); } - columnarBatch = new ColumnarBatch(cols); - columnarBatch.setNumRows(num); - - rowsReturned += num; - numBatched = num; - batchIdx = 0; - pendingCloseKeyVectors = dequeued; - return true; } private void initializeInternal() throws IOException, UnsupportedOperationException { @@ -645,20 +705,18 @@ private void initializeInternal() throws IOException, UnsupportedOperationExcept /** * Sets the storage filter for late materialization. Must be called before {@link #initialize}; - * {@link #initializeLateMaterialization()} (run from {@link #initialize}) inspects the per-file - * schema and decides whether splicing actually engages. + * {@link #initializeLateMaterialization()} then inspects the per-file schema and decides whether + * splicing engages. * - *

Splicing does NOT engage in one real case: all key columns are missing from this physical - * file under schema evolution. The predicate is rewritten with each missing key replaced by the - * constant the reader materializes for it (its existence DEFAULT, else null -- see - * {@code ParquetStorageFilter.rewriteForMissingKeys}) and evaluated as a constant: a true result - * keeps the file with no filtering, a false/null result skips it entirely. Either way the rows - * the scan returns are exactly the rows that satisfy the filter, so this is safe. + *

It does not engage in one real case: every key column is missing from this physical file + * under schema evolution. The predicate is then rewritten with each missing key replaced by the + * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once + * -- true keeps the file unfiltered, false or null skips it. * - *

Every OTHER precondition is guaranteed by planning-time checks in - * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and - * violations throw rather than fall back: once extraction has moved a bloom filter onto the scan - * it is gone from the post-scan Filter, so quietly not applying it would produce wrong rows. + *

Every other precondition is guaranteed by + * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and a + * violation throws rather than falling back: extraction has already removed the filter from the + * post-scan Filter, so not applying it would return rows the query rejected. */ public void setStorageFilter(ParquetStorageFilter storageFilter) { this.storageFilter = storageFilter; @@ -667,10 +725,9 @@ public void setStorageFilter(ParquetStorageFilter storageFilter) { private void initializeLateMaterialization() throws IOException { lateMatReader = reader.getUnderlyingReader(); if (lateMatReader == null) { - // Unreachable in production: the only ParquetRowGroupReader ParquetFileFormat builds is - // ParquetRowGroupReaderImpl, which returns its reader. Dropping the filter here would return - // rows it rejects, since extraction already removed it from the post-scan Filter. - throw new IllegalStateException( + // Unreachable: the only ParquetRowGroupReader ParquetFileFormat builds is + // ParquetRowGroupReaderImpl, which exposes its reader. + throw new UnsupportedFileReadException( "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but " + reader.getClass().getName() + " does not expose one"); } @@ -689,19 +746,16 @@ private void initializeLateMaterialization() throws IOException { for (int i = 0; i < keyIndices.length; i++) { int idx = keyIndices[i]; if (idx < 0 || idx >= parquetColumn.children().size()) { - // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. Fail loudly - // rather than dropping the filter -- extractStorageFilters already removed it from the - // post-scan Filter, so silently ignoring it here would return wrong rows. - throw new IllegalStateException(String.format( + // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. + throw new UnsupportedFileReadException(String.format( "Storage-filter key ordinal %d is out of range for a %d-column requested schema", idx, parquetColumn.children().size())); } ParquetColumn column = parquetColumn.children().apply(idx); if (!column.isPrimitive()) { // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive - // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. Fail loudly for - // the same reason as above. - throw new IllegalStateException( + // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. + throw new UnsupportedFileReadException( "Storage-filter key column is not a primitive Parquet column: " + column.path()); } if (missingColumns.contains(column)) { @@ -729,17 +783,10 @@ private void initializeLateMaterialization() throws IOException { storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues); if (presentKeyColumns.isEmpty()) { - // All key columns missing: the rewritten predicate is fully constant. Evaluate once and - // apply uniformly to the whole file. + // Every key column is missing, so the rewritten predicate is constant for this file. boolean keepAll = storageFilter.evalAllMissing(); - if (keepAll) { - // Predicate is constant-true for this file: no filtering to do. - storageFilter = null; - } else { - // Predicate is constant-false/null: no row from this file can pass the filter. - storageFilter = null; - hitEndOfData = true; - } + storageFilter = null; + hitEndOfData = !keepAll; return; } } @@ -757,14 +804,12 @@ private void initializeLateMaterialization() throws IOException { keySchemaBuilder.addField(requestedSchema.getType(topLevelName)); keyTopLevelNames.add(topLevelName); } - keyOnlyRequestedSchema = keySchemaBuilder.named(requestedSchema.getName()); + requestedColumns = requestedSchema.getColumns(); + keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns(); - // Build the non-key (complement) schema. When the projection has at least one non-key column, - // phase 2 will switch lateMatReader's schema to this and read those columns under finalRanges. - // When the projection is *all* key columns (e.g. a scan whose only column is the bloom probe), - // splicing still pays off: phase 2 has nothing useful to read, so we skip it entirely (no read, - // no IO) and emit batches purely from the key queues. The `nonKeyRequestedSchema != null` check - // downstream gates phase-2 IO. + // Build the non-key (complement) schema, which phase 2 reads under finalRanges. When the + // projection is all key columns, phase 2 has nothing to read and is skipped entirely, which a + // null `nonKeyColumns` is what says downstream. Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage(); int nonKeyFieldCount = 0; for (Type field : requestedSchema.getFields()) { @@ -774,8 +819,7 @@ private void initializeLateMaterialization() throws IOException { } } if (nonKeyFieldCount > 0) { - nonKeyRequestedSchema = nonKeyBuilder.named(requestedSchema.getName()); - requireOffsetIndexesForPhase2(); + nonKeyColumns = nonKeyBuilder.named(requestedSchema.getName()).getColumns(); } initializeSplicingState(presentKeyColumns); @@ -784,60 +828,10 @@ private void initializeLateMaterialization() throws IOException { } /** - * Fails now if any projected column of any row group lacks a Parquet offset index. - * - *

Phase 2 reads a strict subset of a row group's rows, which parquet can only do via the - * offset index; files written before parquet-mr 1.11, or by writers that omit it, have none. We - * cannot - * widen phase 2 to the whole block instead, because the key vectors already hold only the - * survivors and the batch would misalign -- and we cannot skip the filter either, since - * {@code extractStorageFilters} has already removed it from the post-scan Filter. - * - *

Every projected column is checked, not just the non-key ones phase 2 reads, because parquet - * builds one column index store per row group and reuses it. Phase 0 asks for the row ranges - * under the full requested schema, so {@code ColumnIndexStoreImpl.create} is called with the key - * columns in its path set, and it returns its {@code EMPTY} singleton as soon as any one of those - * paths has no offset index. {@code ParquetFileReader.getColumnIndexStore} memoizes that store - * per block, and {@code EMPTY.getOffsetIndex} throws for *every* column. So a key column with no - * offset index kills phase 2 too, with a raw {@code MissingOffsetIndexException} naming some - * non-key column and none of the guidance below. - * - *

Checking up front rather than at the first partially-kept row group is deliberate: whether - * phase 2 needs the offset index otherwise depends on how selective the filter turns out to be on - * this particular file, so the same query would fail or not depending on the data. This is - * conservative -- a filter that happens to keep every row of every block would not have needed - * the offset index -- but such a filter also saves nothing, so failing loudly loses nothing. - * - *

The check itself is free: {@code getOffsetIndexReference()} is a footer field that - * {@link #initialize} has already read. - */ - private void requireOffsetIndexesForPhase2() { - Set projectedPaths = new HashSet<>(); - for (ColumnDescriptor column : requestedSchema.getColumns()) { - projectedPaths.add(ColumnPath.get(column.getPath())); - } - List blocks = lateMatReader.getRowGroups(); - for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) { - for (ColumnChunkMetaData chunk : blocks.get(blockIdx).getColumns()) { - if (projectedPaths.contains(chunk.getPath()) && chunk.getOffsetIndexReference() == null) { - throw new IllegalStateException(String.format( - "Storage-filter pushdown requires a Parquet offset index to read a subset of a row " - + "group, but column %s of row group %d in %s has none. Set %s=false to read " - + "this file.", - chunk.getPath(), blockIdx, lateMatReader.getFile(), - SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key())); - } - } - } - } - - /** - * Populates splicing bookkeeping: {@link #isKeyTopLevel}, {@link #keyVectorQueues}, - * {@link #currentKeyAccumulators}. {@code keyColumnIndices} already index the top-level slots of - * {@link #sparkSchema} (same indexing as {@link #columnVectors}), so they map directly onto - * {@link #isKeyTopLevel}. After this method returns, the splicing path is fully initialized; - * {@link #nextBatch()} and {@link #close()} use {@link #isKeyTopLevel} as the active-state - * indicator. + * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level + * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map + * directly onto {@link #isKeyTopLevel}, which {@link #nextBatch()} and {@link #close()} then + * use as the splicing-is-active indicator. */ @SuppressWarnings("unchecked") private void initializeSplicingState(List presentKeyColumns) { @@ -855,10 +849,25 @@ private void initializeSplicingState(List presentKeyColumns) { currentKeyAccumulators = new WritableColumnVector[numKeys]; currentKeyAccumulatorRowCount = 0; keyCopiers = new ValueCopier[numKeys]; + keyVariableLength = new boolean[numKeys]; + keyFixedBytesPerRow = 0; StructField[] fields = sparkRequestedSchema.fields(); for (int i = 0; i < numKeys; i++) { - keyCopiers[i] = copierFor(fields[keyIndices[i]].dataType()); + DataType dt = fields[keyIndices[i]].dataType(); + keyCopiers[i] = copierFor(dt); + keyVariableLength[i] = isVariableLength(dt); + // One null byte per row either way. A fixed-width value adds its own width, a variable-length + // one the int offset and int length that point at the byte child. + keyFixedBytesPerRow += keyVariableLength[i] ? 1 + 8 : 1 + dt.defaultSize(); + } + } + + /** Whether a key value lives in the vector's byte child rather than in its fixed-width array. */ + private static boolean isVariableLength(DataType dt) { + if (dt instanceof DecimalType decimalType) { + return decimalType.precision() > Decimal.MAX_LONG_DIGITS(); } + return dt instanceof StringType || dt instanceof BinaryType; } /** @@ -937,7 +946,7 @@ private void checkEndOfRowGroup() throws IOException { * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges}, * evaluate the storage filter per row, build {@code finalRanges}. * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}. - * Skipped entirely when {@link #nonKeyRequestedSchema} is null (all-keys projection). + * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection). * * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). * Sets {@link #hitEndOfData} when all row groups have been processed. @@ -951,105 +960,146 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // trip parquet's own `from <= to` assertion. The plain read path skips them too. continue; } - - // Phase 0: rows allowed by the pushed data filter (column-index granularity). Restore the - // full requestedSchema first: phases 1 and 2 below narrow the reader's schema, and - // ParquetFileReader.getRowRanges computes ranges against whatever schema is set (it passes - // the reader's current `paths` to ColumnIndexFilter). This is defensive -- with column-index - // filtering on, getFilteredRecordCount() in initialize() has already memoized every block's - // ranges under the full schema, and with it off we do not call getRowRanges at all. - lateMatReader.setRequestedSchema(requestedSchema); - // ParquetFileReader.getRowRanges only checks whether a filter is pushed, NOT - // options.useColumnIndexFilter(), so calling it unconditionally would keep applying - // column-index filtering after a user turned it off. That conf is the documented escape hatch - // for files whose column index is wrong, and trusting a wrong column index here would drop - // rows for good: finalRanges is a subset of pushedFilterRanges, and the post-scan Filter no - // longer holds this predicate. + // Splicing buffers one key value per surviving row of the whole row group before it can emit + // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it + // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes + // every projected column under finalRanges, key columns included, so nothing is buffered and + // the cost is one extra read of the key columns. + spliceCurrentRowGroup = true; + splicedBytes = 0L; + + // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full + // requestedSchema goes back on first, because phases 1 and 2 narrow it and + // ParquetFileReader.getRowRanges computes ranges against the reader's current paths. + lateMatReader.setRequestedSchema(requestedColumns); + // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(), + // so calling it unconditionally would keep applying column-index filtering after a user + // turned it off -- the documented escape hatch for files whose column index is wrong. + // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of + // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is + // unaffected: it selects pages through the offset index, which this conf says nothing + // about. RowRanges pushedFilterRanges = useColumnIndexFilter ? lateMatReader.getRowRanges(blockIdx) : RowRanges.createSingle(blockRowCount); - if (pushedFilterRanges.rowCount() == 0) { + // RowRanges.rowCount() walks every range, so resolve each range set's count once. + long baselineRows = pushedFilterRanges.rowCount(); + if (baselineRows == 0) { // Pushed data filter rejects this block entirely via column index. Not a storage-filter // skip, so we don't increment storage-filter metrics. continue; } - // Both byte metrics are always wired in production (FileSourceScanLike creates all five - // whenever storageFilters is non-empty), so this only skips the work on the test-only path - // that drives the reader directly. compressedBytesForRowRanges never does IO of its own, so - // there is nothing here to avoid on the production path. + // What this feature can avoid reading is the non-key columns of the rows the storage filter + // rejects, so that is the baseline both byte metrics are measured against: the non-key bytes + // a plain read of this projection would transfer for every row the pushed filter kept. The + // null checks only skip work for a caller that drives this reader without a scan's metrics; + // FileSourceScanLike creates all five whenever storageFilters is non-empty. + // compressedBytesForRowRanges never does IO of its own. StorageFilterMetrics m = storageFilter.metrics(); SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup(); SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering(); boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null; - long baselineRows = pushedFilterRanges.rowCount(); - long baselineBytes = needBytes - ? compressedBytesForRowRanges(lateMatReader, blockIdx, requestedSchema, - pushedFilterRanges) + Map blockChunks = + needBytes ? chunksByPath(lateMatReader, blockIdx) : null; + long nonKeyBaselineBytes = needBytes + ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns, + pushedFilterRanges, baselineRows) : 0L; // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the // storage filter per row. - lateMatReader.setRequestedSchema(keyOnlyRequestedSchema); - long phase1Bytes = needBytes - ? compressedBytesForRowRanges(lateMatReader, blockIdx, keyOnlyRequestedSchema, - pushedFilterRanges) - : 0L; + lateMatReader.setRequestedSchema(keyOnlyColumns); PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges); if (keyPages == null) { // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already // know pushedFilterRanges selects at least one row. Skipping the block here would drop its // surviving rows from the output, so assert rather than `continue`. - throw new IllegalStateException( - "No key pages for row group " + blockIdx + " despite " - + pushedFilterRanges.rowCount() + " rows selected by the pushed filter"); + throw new UnsupportedFileReadException( + "No key pages for row group " + blockIdx + " despite " + baselineRows + + " rows selected by the pushed filter"); } RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges); + long finalRowCount = finalRanges.rowCount(); - if (finalRanges.rowCount() == 0) { - // Every surviving row was rejected by the storage filter; skip block entirely. We still - // paid phase-1 to read the key column, so the bytes avoided vs a no-storage-filter read - // are baseline - phase1 (the non-key bytes the no-filter path would have read). + if (finalRowCount == 0) { + // Every surviving row was rejected by the storage filter; skip block entirely, which avoids + // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is + // not part of the baseline, so nothing has to be subtracted from it here. SQLMetric rgSkipped = m.rowGroupsSkipped(); if (rgSkipped != null) rgSkipped.add(1L); SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup(); if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows); - if (bytesAvoidedRg != null) bytesAvoidedRg.add(baselineBytes - phase1Bytes); + if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes); continue; } // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely - // when the projection is all keys (nonKeyRequestedSchema is null); emit reconstructs each - // batch from the key queues alone. + // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from + // the key queues alone. long keptRows; long phase2Bytes; PageReadStore dataPages = null; - if (nonKeyRequestedSchema == null) { - keptRows = finalRanges.rowCount(); + // An all-keys projection has nothing for phase 2 to read -- but only if the key values were + // buffered. A row group past the cap has to read them here like any other column. + if (nonKeyColumns == null && spliceCurrentRowGroup) { + keptRows = finalRowCount; phase2Bytes = 0L; } else { - lateMatReader.setRequestedSchema(nonKeyRequestedSchema); - // requireOffsetIndexesForPhase2() already established that every projected column of every - // row group has an offset index, so this page-filtering read cannot fail for want of one. - dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); + lateMatReader.setRequestedSchema( + spliceCurrentRowGroup ? nonKeyColumns : requestedColumns); + // Reading a strict subset of a block's rows needs a Parquet offset index, and parquet + // enforces that itself: it resolves every requested column's offset index before reading + // anything, and a column without one makes its column index store throw + // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that + // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), + // have none. Degrading to a whole-block read is not an option, because the key vectors + // hold only the survivors and the batch would misalign, and neither is dropping the + // predicate, which extraction has removed from the post-scan Filter. So the read fails, + // and all this adds is what the user can do about it. + // + // Nothing is checked up front: a row group the filter keeps whole never needs the index + // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and + // one it rejects whole is never read at all, so a file with no page index still scans as + // long as the filter never has to prune inside a row group. + try { + dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); + } catch (MissingOffsetIndexException e) { + throw new UnsupportedFileReadException(String.format( + "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its " + + "filter kept in row group %d of %s, but the file was written without a page " + + "index. Set %s=false to read this file.", + finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(), + SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e); + } if (dataPages == null) { // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges, // both excluded above. Match phase 1 and fail with a message rather than an NPE. - throw new IllegalStateException( - "No data pages for row group " + blockIdx + " despite " + finalRanges.rowCount() + throw new UnsupportedFileReadException( + "No data pages for row group " + blockIdx + " despite " + finalRowCount + " surviving rows"); } keptRows = dataPages.getRowCount(); - phase2Bytes = bytesAvoidedPf != null - ? compressedBytesForRowRanges(lateMatReader, blockIdx, nonKeyRequestedSchema, - finalRanges) - : 0L; + // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`. + if (needBytes && bytesAvoidedPf != null) { + phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, + nonKeyColumns, finalRanges, finalRowCount); + if (!spliceCurrentRowGroup) { + // This row group gave splicing up, so phase 2 read the key columns a second time. The + // baseline counts them once, in phase 1, so the extra read is a cost against it -- + // which can make the row group's contribution negative, and that is the truth about it. + phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, + keyOnlyColumns, finalRanges, finalRowCount); + } + } else { + phase2Bytes = 0L; + } } long filteredRows = baselineRows - keptRows; SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup(); if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows); if (bytesAvoidedPf != null) { - bytesAvoidedPf.add(baselineBytes - phase1Bytes - phase2Bytes); + bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes); } if (dataPages != null) { @@ -1057,7 +1107,7 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { rowIndexGenerator.initFromPageReadStore(dataPages); } for (int i = 0; i < columnVectors.length; i++) { - if (isKeyTopLevel[i]) { + if (spliceCurrentRowGroup && isKeyTopLevel[i]) { // Key columns are sourced from the queues during emit; skip phase-2 reader init. continue; } @@ -1070,10 +1120,25 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { hitEndOfData = true; } + /** - * Compressed bytes the reader transfers for the leaf columns of {@code schema} when it reads - * exactly {@code rowRanges} of the given block. Page headers and the dictionary page are - * included, because both are read whenever any page of a chunk is read. + * The block's column chunks by path, built once per row group and shared by the byte-metric calls + * that consume it, since {@link BlockMetaData} offers no lookup of its own. + */ + private static Map chunksByPath( + ParquetFileReader reader, int blockIndex) { + Map chunks = new HashMap<>(); + for (ColumnChunkMetaData chunk : reader.getRowGroups().get(blockIndex).getColumns()) { + chunks.put(chunk.getPath(), chunk); + } + return chunks; + } + + /** + * Compressed bytes the reader transfers for the given leaf {@code columns} when it reads exactly + * {@code rowRanges} of the given block. Page headers and the dictionary page are included, since + * both are read whenever any page of a chunk is read. {@code rowRangeCount} is + * {@code rowRanges.rowCount()}, passed in because that walks every range and the caller has it. * *

Two sources, chosen so this never causes IO of its own: *

    @@ -1088,27 +1153,23 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { *
* *

Columns absent from this physical file (schema evolution) contribute nothing, which is - * correct: the reader transfers nothing for them. Every caller for a given block walks the same - * metadata, so a skipped column drops out of the baseline and the per-phase totals alike. + * correct: the reader transfers nothing for them. */ private static long compressedBytesForRowRanges( ParquetFileReader reader, int blockIndex, - MessageType schema, - RowRanges rowRanges) { - if (schema == null || rowRanges.rowCount() == 0 || schema.getColumns().isEmpty()) { + Map chunks, + List columns, + RowRanges rowRanges, + long rowRangeCount) { + if (columns == null || columns.isEmpty() || rowRangeCount == 0) { return 0L; } - BlockMetaData block = reader.getRowGroups().get(blockIndex); - long blockRowCount = block.getRowCount(); - Map chunks = new HashMap<>(); - for (ColumnChunkMetaData chunk : block.getColumns()) { - chunks.put(chunk.getPath(), chunk); - } - boolean wholeBlock = rowRanges.rowCount() == blockRowCount; + long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount(); + boolean wholeBlock = rowRangeCount == blockRowCount; ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex); long total = 0L; - for (ColumnDescriptor column : schema.getColumns()) { + for (ColumnDescriptor column : columns) { ColumnPath path = ColumnPath.get(column.getPath()); ColumnChunkMetaData chunk = chunks.get(path); if (chunk == null) { @@ -1155,16 +1216,15 @@ private static long dictionaryPageSize(ColumnChunkMetaData chunk) { } /** - * Reads all rows of the given key-only {@link PageReadStore} (which contains only rows in - * {@code pushedFilterRanges}) in capacity-sized chunks, evaluates the storage filter on each row, - * builds a {@link RowRanges} of surviving rows in original block-row coordinates, and appends - * survivor key values into the per-key-column accumulators ({@link #currentKeyAccumulators}). - * When an accumulator hits {@link #capacity}, it's pushed into {@link #keyVectorQueues} and a - * fresh one is allocated. After all rows have been examined, any partial trailing accumulator is - * pushed too. + * Evaluates the storage filter over every row of a key-only {@link PageReadStore}, in + * capacity-sized chunks, and returns the surviving rows as {@link RowRanges} in block-row + * coordinates. The result is a subset of {@code pushedFilterRanges}: rows outside it were never + * read. * - *

The result is a subset of {@code pushedFilterRanges}: rows not in {@code - * pushedFilterRanges} were never read and are implicitly excluded. + *

Each survivor's key values are appended to {@link #currentKeyAccumulators} for the emit path + * to splice, until the buffer passes its cap. From there the row group is evaluated without + * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns + * again along with everything else. */ private RowRanges evaluateStorageFilter( PageReadStore keyPages, @@ -1178,10 +1238,12 @@ private RowRanges evaluateStorageFilter( } ensureCurrentKeyAccumulatorsAllocated(); - long keyRowsTotal = pushedFilterRanges.rowCount(); PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator(); RowRanges.Builder finalRangesBuilder = RowRanges.builder(); - long remaining = keyRowsTotal; + // Recomputed rather than taken from the caller: a count that disagreed with this iterator would + // silently drop surviving rows, and no post-scan Filter is left to catch that. + long remaining = pushedFilterRanges.rowCount(); + boolean accumulate = true; while (remaining > 0) { int num = (int) Math.min((long) capacity, remaining); for (int i = 0; i < keyScratchVectors.length; i++) { @@ -1193,19 +1255,25 @@ private RowRanges evaluateStorageFilter( long blockRow = rowIndexIter.nextLong(); if (storageFilter.test(keyScratchBatch.getRow(r))) { finalRangesBuilder.addSelectedRow(blockRow); - appendSurvivorRowToAccumulators(r); + if (accumulate) { + accumulate = appendSurvivorRowToAccumulators(r); + } } } remaining -= num; } - finalizePartialAccumulators(); + if (accumulate) { + finalizePartialAccumulators(); + } return finalRangesBuilder.build(); } private void ensureKeyScratchAllocated() { if (keyScratchVectors != null) return; + // Assigned before the loop on purpose: an allocation failure part way through then leaves the + // vectors allocated so far reachable for `close()`, which walks this array element-wise. keyScratchVectors = new WritableColumnVector[keyDescriptors.length]; boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP; int[] keyIndices = storageFilter.keyColumnIndices(); @@ -1219,9 +1287,8 @@ private void ensureKeyScratchAllocated() { } /** - * Allocates the per-key-column accumulator vectors if any slot is null (i.e. the previous - * accumulator was just pushed to the queue or this is the first row group). Each accumulator has - * {@link #capacity} rows. + * Allocates any accumulator slot left null by the last push to the queues, {@link #capacity} rows + * each. */ private void ensureCurrentKeyAccumulatorsAllocated() { boolean useOffHeap = MEMORY_MODE == MemoryMode.OFF_HEAP; @@ -1238,16 +1305,20 @@ private void ensureCurrentKeyAccumulatorsAllocated() { } /** - * Appends row {@code srcRow} of each {@link #keyScratchVectors} into the corresponding - * {@link #currentKeyAccumulators}. When the accumulators fill, they're pushed onto their queues - * and fresh ones allocated. All key columns are appended in lockstep so accumulators stay - * aligned. + * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their + * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned. + * + *

Returns false when the buffered survivors have passed + * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has + * given splicing up, in which case it has already released what it held and the caller must stop + * calling this. */ - private void appendSurvivorRowToAccumulators(int srcRow) { + private boolean appendSurvivorRowToAccumulators(int srcRow) { final int dstRow = currentKeyAccumulatorRowCount; final WritableColumnVector[] accs = currentKeyAccumulators; final WritableColumnVector[] srcs = keyScratchVectors; final ValueCopier[] copiers = keyCopiers; + long valueBytes = 0L; for (int i = 0, n = accs.length; i < n; i++) { WritableColumnVector src = srcs[i]; WritableColumnVector dst = accs[i]; @@ -1255,16 +1326,44 @@ private void appendSurvivorRowToAccumulators(int srcRow) { dst.putNull(dstRow); } else { copiers[i].copy(dst, dstRow, src, srcRow); + // Measured on the destination: a dictionary-encoded source has no length of its own, since + // its values are read through the dictionary. + if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow); } } + splicedBytes += keyFixedBytesPerRow + valueBytes; currentKeyAccumulatorRowCount = dstRow + 1; if (currentKeyAccumulatorRowCount == capacity) { for (int i = 0; i < currentKeyAccumulators.length; i++) { keyVectorQueues[i].addLast(currentKeyAccumulators[i]); currentKeyAccumulators[i] = null; } + // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A + // row group whose survivors fit in a single accumulator is never checked at all: it then + // holds one capacity-sized vector per key column, which is what a plain read holds anyway. + if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) { + abandonSplicing(); + return false; + } ensureCurrentKeyAccumulatorsAllocated(); } + return true; + } + + /** + * Gives up splicing for the row group being evaluated and releases every survivor vector it has + * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent + * batch, so the rows are unaffected. + */ + private void abandonSplicing() { + for (java.util.ArrayDeque q : keyVectorQueues) { + for (WritableColumnVector v : q) v.close(); + q.clear(); + } + closeAll(currentKeyAccumulators); + Arrays.fill(currentKeyAccumulators, null); + currentKeyAccumulatorRowCount = 0; + spliceCurrentRowGroup = false; } /** @@ -1281,17 +1380,11 @@ private void finalizePartialAccumulators() { } /** - * Closes anything held by the splicing path: pending dequeued key vectors not yet rolled over, - * any vectors still queued (e.g. on early termination), and partially-filled accumulators. - * Called from {@link #close()}. + * Closes anything held by the splicing path: every vector still queued, the published head + * included, and partially-filled accumulators. Called from {@link #close()}. */ private void closeSplicingState() { - if (pendingCloseKeyVectors != null) { - for (WritableColumnVector v : pendingCloseKeyVectors) { - if (v != null) v.close(); - } - pendingCloseKeyVectors = null; - } + keyVectorsPublished = false; if (keyVectorQueues != null) { for (java.util.ArrayDeque q : keyVectorQueues) { if (q != null) { @@ -1300,18 +1393,21 @@ private void closeSplicingState() { } } } - if (currentKeyAccumulators != null) { - for (WritableColumnVector v : currentKeyAccumulators) { - if (v != null) v.close(); - } - currentKeyAccumulators = null; + closeAll(currentKeyAccumulators); + currentKeyAccumulators = null; + } + + /** Closes every non-null vector of {@code vectors}; tolerates a null array. */ + private static void closeAll(WritableColumnVector[] vectors) { + if (vectors == null) return; + for (WritableColumnVector v : vectors) { + if (v != null) v.close(); } } /** - * Per-key-column value copier: appends one value from {@code src[srcRow]} to - * {@code dst[dstRow]}. Picked once at init via {@link #copierFor(DataType)}; called per surviving - * row in {@link #appendSurvivorRowToAccumulators}. Caller handles null sources. + * Copies one key value between column vectors. Picked per key column at init time by + * {@link #copierFor(DataType)}; the caller handles null sources. */ @FunctionalInterface private interface ValueCopier { @@ -1363,13 +1459,11 @@ private static ValueCopier copierFor(DataType dt) { } return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow)); } - if (dt instanceof StringType - || dt instanceof VarcharType - || dt instanceof CharType - || dt instanceof BinaryType) { + // StringType covers CHAR and VARCHAR: both extend it. + if (dt instanceof StringType || dt instanceof BinaryType) { return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow)); } - throw new UnsupportedOperationException( + throw new UnsupportedFileReadException( "Splicing storage-filter pushdown does not support key type: " + dt); } @@ -1399,28 +1493,20 @@ private void initColumnReader(PageReadStore pages, ParquetColumnVector cv) throw * use `OnHeapColumnVector` when `useOffHeap` is false, the constant columns * always use `ConstantColumnVector`. * - *

Data slots whose indices appear in {@code skipDataSlots} are left null. The splicing - * late-materialization path uses this to skip allocation for storage-filter key columns: those - * slots are sourced from per-key-column queues populated in phase 1 (see {@code - * nextBatchSplicing}). - * * Capacity is the initial capacity of the vector, and it will grow as necessary. * Capacity is in number of elements, not number of bytes. */ private ColumnVector[] allocateColumns( - int capacity, StructType schema, boolean useOffHeap, int constantColumnLength, - Set skipDataSlots) { + int capacity, StructType schema, boolean useOffHeap, int constantColumnLength) { StructField[] fields = schema.fields(); int fieldsLength = fields.length; ColumnVector[] vectors = new ColumnVector[fieldsLength]; if (useOffHeap) { for (int i = 0; i < fieldsLength - constantColumnLength; i++) { - if (skipDataSlots != null && skipDataSlots.contains(i)) continue; vectors[i] = new OffHeapColumnVector(capacity, fields[i].dataType()); } } else { for (int i = 0; i < fieldsLength - constantColumnLength; i++) { - if (skipDataSlots != null && skipDataSlots.contains(i)) continue; vectors[i] = new OnHeapColumnVector(capacity, fields[i].dataType()); } } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index 67a76449bfb95..6b281c9e8dbc6 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -303,11 +303,9 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { // Filters on non-partition columns. def dataFilters: Seq[Expression] - // Filters that should be evaluated lazily by the storage layer (e.g. parquet reader) for IO - // pruning of value columns based on key column evaluation. These may reference subqueries (e.g. a - // runtime bloom filter built from a join build side) and are materialized at task launch time. - // Defaults to Nil so that a scan which does not support storage-filter pushdown need not know - // about it. + // Filters the storage layer evaluates to prune value-column IO based on key-column evaluation. + // These may reference subqueries (e.g. a runtime bloom filter built from a join build side) and + // are materialized at task launch time. Nil for a scan that does not support pushing them. def storageFilters: Seq[Expression] = Nil // Disable bucketed scan based on physical query plan, see rule // [[DisableUnnecessaryBucketedScan]] for details. @@ -660,8 +658,7 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { } ++ storageFilterMetrics ++ driverMetrics protected lazy val storageFilterMetrics: Map[String, SQLMetric] = if (storageFilters.nonEmpty) { - // A row group is skipped, a row is excluded, a byte is avoided. See StorageFilterMetrics for - // why each counter uses its own verb. + // See `StorageFilterMetrics` for what each of these counts. Map( FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED -> SQLMetrics.createMetric(sparkContext, "row groups skipped by storage filter"), @@ -799,11 +796,10 @@ case class FileSourceScanExec( lazy val inputRDD: RDD[InternalRow] = { val options = relation.options + (FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString) - // Only route through the storage-filter entry point when there is something to push. A - // `FileFormat` subclass that customizes reading by overriding `buildReaderWithPartitionValues` - // -- the long-standing entry point -- would otherwise be bypassed on every query, because - // `ParquetFileFormat` overrides `buildReaderWithStorageFilters` with a full reader - // implementation that the subclass knows nothing about. + // Only route through the storage-filter entry point when there is something to push, so that a + // `FileFormat` subclass which customizes reading by overriding `buildReaderWithPartitionValues` + // keeps being used on every other query: `ParquetFileFormat` answers + // `buildReaderWithStorageFilters` with a full reader the subclass knows nothing about. val readFile: (PartitionedFile) => Iterator[InternalRow] = if (preparedStorageFilters.isEmpty) { relation.fileFormat.buildReaderWithPartitionValues( @@ -844,17 +840,13 @@ case class FileSourceScanExec( if (storageFilters.isEmpty) { Nil } else { - // Trust the planning-time decision: when [[FileSourceStrategy.extractStorageFilters]] moved a - // bloom filter into [[storageFilters]], it removed that conjunct from the post-scan Filter. - // Re-checking the conf here would silently drop the filter if the user toggled it off between - // planning and execution, producing wrong results. The conf only gates whether extraction - // happens at planning time. + // No conf check here: extraction has already removed these conjuncts from the post-scan + // Filter, so re-reading the conf would drop the filter for good if the user turned it off + // between planning and execution. The conf gates extraction at planning time only. // - // `output` is constructed by FileSourceStrategy as - // `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++ - // constantMetadataColumns` - // and `requiredSchema` is the StructType of the first two groups, so the first - // `requiredSchema.length` attributes of `output` correspond 1:1 to requiredSchema fields. + // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++ + // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so + // the first `requiredSchema.length` attributes line up with its fields. val requestedDataAttrs = output.take(requiredSchema.length) storageFilters.map { expr => val subqueryReplaced = expr.transform { case s: execution.ScalarSubquery => s.toLiteral } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala index 2eb4d04c7d897..8662ce7ebd03b 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala @@ -269,7 +269,15 @@ object DataSourceUtils extends PredicateHelper { QueryExecutionErrors.sparkUpgradeInWritingDatesError(format, config) } + /** + * Whether `ignoreCorruptFiles` may swallow this failure and skip the rest of its file. + * + * [[UnsupportedFileReadException]] is excluded because it does not report a corrupt file: it says + * the file cannot support a read the plan depends on. Skipping the rest of such a file would drop + * rows that are perfectly readable, and would do it silently. + */ def shouldIgnoreCorruptFileException(e: Throwable): Boolean = e match { + case _: UnsupportedFileReadException => false case _: RuntimeException | _: IOException | _: InternalError => true case _ => false } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala index 6503af3636ddf..9a20bdcd37928 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala @@ -172,15 +172,14 @@ trait FileFormat { * on key-column evaluation (e.g., late materialization with a runtime bloom filter). * * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no - * storage filter at all. File formats that support storage-filter pushdown (e.g., Parquet) should - * override this method. + * storage filter at all. A format that supports storage-filter pushdown overrides this, and must + * not let the two builders call each other: this default delegates one way, so an override that + * delegates back recurses until the driver's stack runs out, `super` included, since that call is + * virtual too. Route both to a private implementation instead, the way `ParquetFileFormat` does. * * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than - * dropping it. The planner removes an extracted conjunct from the post-scan `Filter`, so a reader - * that ignores it returns rows the filter rejects. Every other layer of the feature fails loudly - * for the same reason. Unreachable today, because `FileSourceStrategy.extractStorageFilters` only - * extracts for `ParquetFileFormat` itself, but that keeps the invariant one method away from the - * code that relies on it. + * dropping it: the planner removes an extracted conjunct from the post-scan `Filter`, so a reader + * that ignores it returns rows the filter rejects. * * Scalar subqueries inside `storageFilters` are expected to have been materialized before this * method is called, so that the returned reader can be safely serialized to executors. @@ -207,6 +206,17 @@ trait FileFormat { sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf) } + /** + * Whether this format's reader can evaluate `expr` as a storage filter, i.e. whether the planner + * may extract it from the post-scan `Filter` and hand it to [[buildReaderWithStorageFilters]]. + * + * The planner decides what it can see from the plan -- that the conjunct is deterministic and + * references only projected data columns -- and asks this for everything else, so the expression + * shapes and column types a reader supports stay in that reader's own package. A format that + * answers true for an expression must be able to honor it: the planner removes it from the plan. + */ + def supportsStorageFilter(expr: Expression): Boolean = false + /** * Create a file metadata struct column containing fields supported by the given file format. */ diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala index a087bc64a4a4a..db92448468f9f 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala @@ -33,7 +33,6 @@ import org.apache.spark.sql.catalyst.trees.TreePattern.{PLAN_EXPRESSION, SCALAR_ import org.apache.spark.sql.catalyst.types.DataTypeUtils import org.apache.spark.sql.classic.Strategy import org.apache.spark.sql.execution.{FileSourceScanExec, SparkPlan} -import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, ParquetStorageFilter} import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types.{DoubleType, FloatType, StructType} import org.apache.spark.util.ArrayImplicits._ @@ -153,30 +152,32 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { } /** - * Splits `afterScanFilters` into bloom-filter conjuncts that can be pushed to the storage layer - * for late materialization (returned as the first element) and the remaining filters that stay as - * a post-scan FilterExec (returned as the second element). + * Splits `afterScanFilters` into conjuncts the file format can evaluate at the storage layer for + * late materialization (returned as the first element) and the remaining filters that stay as a + * post-scan FilterExec (returned as the second element). * - * Eligibility (statically checked here so the runtime never silently loses the filter): + * Everything is checked here, statically, so the runtime never silently loses a filter. Two of + * the conditions are per scan, and failing either leaves every filter in the plan: * - The storage-filter pushdown SQL conf is on. - * - The file format is exactly [[ParquetFileFormat]]. Subclasses are excluded on purpose: they - * may customize reading by overriding `buildReaderWithPartitionValues`, and attaching storage - * filters would route the scan through `ParquetFileFormat`'s own reader instead, silently - * dropping whatever the subclass does. - * - The vectorized reader is feasible for the schema the reader will actually see, i.e. - * `partitionSchema ++ outputDataSchema` -- the same schema `ParquetFileFormat.buildReader` - * derives `enableVectorizedReader` from. - * - The conjunct is a top-level [[BloomFilterMightContain]] (not nested under OR/NOT). - * - The conjunct is deterministic. `ParquetStorageFilter.test` evaluates the predicate without - * calling `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a - * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. No - * such bloom exists today, because the only producer is `InjectRuntimeFilter` and a join key - * is deterministic, but this gate should not depend on a distant rule. - * - The bloom's value-side references are projected data columns whose type the reader's value - * copier supports (see [[ParquetStorageFilter.isSupportedKeyType]]). + * - [[FileFormat.supportBatch]] holds for the schema the reader will see, + * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives + * its own vectorized-read decision from. Late materialization needs a batch read, so this asks + * about batch support rather than naming a format. * - * If any condition fails, ALL bloom filters stay in the second element to preserve the existing - * fallback behavior. + * The rest are per conjunct, and one that fails any of them stays in the post-scan Filter: + * - It is deterministic. A reader evaluates the predicate without + * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a + * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. + * - It references at least one column, and every column it references is a projected data + * column. A reference to something the scan does not read cannot be evaluated by the reader. + * - [[FileFormat.supportsStorageFilter]] accepts it. That is where the expression shapes and + * column types a reader can evaluate live, so this method names neither a format nor a type. + * + * One last condition is on the set that survives: at least one projected data column must be left + * for the reader to prune. A scan that projects nothing but the filter's own key columns reads + * the same columns for the same rows either way, since the reader has to read a key column to + * evaluate the filter on it, so pushing can only add the cost of evaluating the predicate outside + * the generated code. Extraction is dropped entirely in that case. */ private def extractStorageFilters( afterScanFilters: ExpressionSet, @@ -186,25 +187,20 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { val sparkSession = fsRelation.sparkSession val sqlConf = sparkSession.sessionState.conf if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters) - if (fsRelation.fileFormat.getClass != classOf[ParquetFileFormat]) { - return (Nil, afterScanFilters) - } - // Mirror the runtime check for vectorized read feasibility. Storage filters drive late - // materialization in the vectorized parquet reader; without it, the scan would silently - // ignore them. val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields) if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) { return (Nil, afterScanFilters) } val dataAttrs = AttributeSet(readDataColumns) - val (eligible, rest) = afterScanFilters.partition { - case bloom: BloomFilterMightContain => - val refs = bloom.valueExpression.references - bloom.deterministic && refs.nonEmpty && refs.forall { a => - dataAttrs.contains(a) && ParquetStorageFilter.isSupportedKeyType(a.dataType) - } - case _ => false + val (eligible, rest) = afterScanFilters.partition { expr => + val refs = expr.references + expr.deterministic && refs.nonEmpty && refs.forall(dataAttrs.contains) && + fsRelation.fileFormat.supportsStorageFilter(expr) + } + val keyAttrs = AttributeSet(eligible.toSeq.flatMap(_.references)) + if (eligible.isEmpty || readDataColumns.forall(keyAttrs.contains)) { + return (Nil, afterScanFilters) } (eligible.toSeq, ExpressionSet(rest)) } @@ -276,10 +272,10 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { logInfo(log"Post-Scan Filters: ${MDC(POST_SCAN_FILTERS, afterScanFilters.simpleString(maxToStringFields))}") - // `filterAttributes` is deliberately computed from `afterScanFilters` *before* storage-filter - // extraction (which happens further down, once `outputDataSchema` is known), so a column - // referenced only by an extracted bloom filter is still part of `requiredAttributes` and - // survives projection pruning -- the reader needs to read it to evaluate the filter. + // `filterAttributes` is computed from `afterScanFilters` before storage-filter extraction, + // which happens further down once `outputDataSchema` is known, so a column referenced only by + // an extracted filter is still in `requiredAttributes` and survives projection pruning. The + // reader has to read it to evaluate the filter. val filterAttributes = AttributeSet(afterScanFilters ++ stayUpFilters) val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects val requiredAttributes = AttributeSet(requiredExpressions) @@ -357,12 +353,9 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { val outputDataSchema = (readDataColumns ++ generatedMetadataColumns).toStructType - // Extract bloom-filter conjuncts that can be pushed to the storage layer for late - // materialization. Eligible bloom filters become `storageFilters` on the scan and are dropped - // from the post-scan Filter (the late-mat path produces exact output). Ineligible ones stay - // in `afterScanFilters` as the existing fallback. This runs here, rather than next to - // `afterScanFilters`, because eligibility depends on `outputDataSchema` -- the schema the - // reader will actually see, and hence what its vectorized-read feasibility is decided from. + // Extracted conjuncts become `storageFilters` on the scan and leave the post-scan Filter, + // since the reader applies them exactly. This runs here rather than next to + // `afterScanFilters` because eligibility depends on `outputDataSchema`. val (storageFilters, remainingAfterScanFilters) = extractStorageFilters( afterScanFilters, fsRelation, readDataColumns, outputDataSchema) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala new file mode 100644 index 0000000000000..cabc5e416c2f5 --- /dev/null +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.execution.datasources + +/** + * Thrown when a file cannot support a read the plan depends on, as opposed to being corrupt or + * missing. The distinction matters because `ignoreCorruptFiles` asks Spark to skip the rest of a + * file whose read failed, and [[DataSourceUtils.shouldIgnoreCorruptFileException]] excludes this + * exception for that reason: skipping here would silently drop rows of a perfectly good file. + * + * A reader throws this when the plan above it assumes the reader does something the file cannot + * express. Storage-filter pushdown is the case that motivated it: the planner removes the pushed + * conjunct from the post-scan `Filter`, so a reader that cannot apply it has to fail the query + * rather than return rows the filter rejects. + */ +class UnsupportedFileReadException(message: String, cause: Throwable) + extends RuntimeException(message, cause) { + + def this(message: String) = this(message, null) +} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index fa33f0d1bd307..0118fb8e1aeb9 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -192,11 +192,20 @@ class ParquetFileFormat filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = { - buildReaderWithStorageFilters( + buildParquetReader( sparkSession, dataSchema, partitionSchema, requiredSchema, filters, Nil, options, hadoopConf, Map.empty) } + /** + * Subclasses answer false on purpose, even though they inherit this reader: a subclass may + * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage + * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass + * whatever the subclass does. + */ + override def supportsStorageFilter(expr: Expression): Boolean = + getClass == classOf[ParquetFileFormat] && ParquetStorageFilter.isSupportedStorageFilter(expr) + override def buildReaderWithStorageFilters( sparkSession: SparkSession, dataSchema: StructType, @@ -207,6 +216,24 @@ class ParquetFileFormat options: Map[String, String], hadoopConf: Configuration, storageFilterMetrics: Map[String, SQLMetric]): PartitionedFile => Iterator[InternalRow] = { + buildParquetReader(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, + storageFilters, options, hadoopConf, storageFilterMetrics) + } + + /** + * The implementation behind both public entry points above, which is why neither of them calls + * the other -- see the warning on `FileFormat.buildReaderWithStorageFilters`. + */ + private def buildParquetReader( + sparkSession: SparkSession, + dataSchema: StructType, + partitionSchema: StructType, + requiredSchema: StructType, + filters: Seq[Filter], + storageFilters: Seq[Expression], + options: Map[String, String], + hadoopConf: Configuration, + storageFilterMetrics: Map[String, SQLMetric]): PartitionedFile => Iterator[InternalRow] = { val sqlConf = getSqlConf(sparkSession) setupHadoopConf(hadoopConf, sqlConf, requiredSchema) @@ -247,18 +274,16 @@ class ParquetFileFormat val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val archiveFormatEnabled = parquetOptions.archiveFormatEnabled - // A non-empty `storageFilters` means `FileSourceStrategy.extractStorageFilters` already removed - // those conjuncts from the post-scan Filter, so there is no longer anything else in the plan - // that would apply them. Quietly not installing them here would return extra rows, so anything - // that stops us from honoring them has to fail loudly instead. - // - // `enableVectorizedReader` is recomputed from the live session conf when the RDD is built, i.e. - // after planning, so flipping spark.sql.parquet.enableVectorizedReader (or the nested-column - // variant) between planning and execution lands here. + // Extraction has already removed these conjuncts from the post-scan Filter, so nothing else in + // the plan would apply them: anything that stops the reader from honoring them fails loudly + // rather than dropping them. `enableVectorizedReader` is one such thing, and it is recomputed + // from the live session conf when the RDD is built, so a flip of + // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands + // here. val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) { None } else if (!enableVectorizedReader) { - throw new IllegalStateException( + throw new UnsupportedFileReadException( "Cannot honor storage filters " + storageFilters.mkString("[", ", ", "]") + " because the " + "vectorized Parquet reader is disabled for schema " + resultSchema.catalogString + ". " + @@ -279,7 +304,8 @@ class ParquetFileFormat FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null)) // `create` requires every condition extractStorageFilters already pre-checked, so it throws // rather than letting us drop the filter. - Some(ParquetStorageFilter.create(storageFilters, requiredSchema, metrics)) + Some(ParquetStorageFilter.create(storageFilters, requiredSchema, metrics, + sqlConf.parquetStorageFilterPushdownMaxSplicedRowGroupBytes)) } // Should always be set by FileSourceScanExec creating this. diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala index d37f815921f69..b578ba8d0042c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala @@ -18,34 +18,30 @@ package org.apache.spark.sql.execution.datasources.parquet import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BoundReference, Expression, Literal, Predicate} +import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate} import org.apache.spark.sql.execution.metric.SQLMetric import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType} /** - * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. All counters - * are scoped to what the storage filter added on top of a no-storage-filter read of the same - * projection. All fields are nullable; a null field disables that metric. - * - * Each counter names its own quantity, so the three verbs are deliberate. A row group is *skipped*, - * meaning its data columns were never read, though phase 1 did read its key columns. A row is - * *excluded*, meaning it never reached the output. A byte is *avoided*, meaning it was never - * transferred. - * - * The row counters' suffix says *where* the row was excluded, not by which mechanism: a row inside - * a kept row group is read as part of its page and dropped during decode, so page filtering did not - * save it. On an all-keys projection there is no page filtering at all, and - * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped. + * Optional SQL metrics the reader updates while applying a [[ParquetStorageFilter]]. Every + * counter is scoped to what the storage filter added on top of a read of the same projection + * without one. All fields are nullable; a null field disables that metric. * * - [[rowGroupsSkipped]] counts row groups whose data columns were never read. - * - [[rowsExcludedByRowGroup]] sums rows excluded by full row-group skips (per skipped block, the - * count of rows that survived the pushed data filter). + * - [[rowsExcludedByRowGroup]] sums the rows those skips excluded, per skipped block the rows that + * survived the pushed data filter. * - [[rowsExcludedWithinRowGroup]] sums rows excluded inside row groups that were kept. - * - [[bytesAvoidedByRowGroup]] sums `baseline - phase1` for skipped row groups (phase 1 still - * reads the key column on every block, so the savings are the non-key bytes the no-filter path - * would have read; zero on all-keys-projection scans). - * - [[bytesAvoidedByPageFiltering]] sums `baseline - phase1 - phase2` for kept row groups: the - * non-key bytes pruned by `finalRanges` page selection beyond what phase 1 already read. + * - [[bytesAvoidedByRowGroup]] sums, per skipped row group, the non-key bytes a plain read of this + * projection would have transferred for the rows that survived the pushed data filter. Phase 1 + * reads the key columns of every block, so key bytes are never part of it, and it is zero on an + * all-keys projection, which can avoid nothing. + * - [[bytesAvoidedByPageFiltering]] sums, per kept row group, that same non-key baseline minus the + * bytes phase 2 read, which is what `finalRanges` page selection pruned. + * + * The row counters' suffix says where a row was excluded, not what would have saved it: a row + * inside a kept row group is read as part of its page and dropped during decode, so page + * filtering did not save it. An all-keys projection has no page filtering at all, and + * [[rowsExcludedWithinRowGroup]] still counts every row the filter dropped. */ case class StorageFilterMetrics( rowGroupsSkipped: SQLMetric = null, @@ -63,16 +59,12 @@ case class StorageFilterMetrics( * columns referenced by the filter. [[boundExpression]] has its references rewritten to * [[BoundReference]]s pointing at positions 0..(keyColumnIndices.length - 1); the reader must * evaluate it against rows whose fields correspond to those key columns in that order. - * - * [[metrics]] carries the optional SQL metrics the reader updates as it applies the filter (always - * non-negative under splicing: phase 1 reads only the key columns, phase 2 reads only non-key - * columns under surviving row ranges, so the avoided bytes are exactly the non-key bytes the - * no-filter path would have read but we didn't). */ class ParquetStorageFilter private ( val keyColumnIndices: Array[Int], val boundExpression: Expression, - val metrics: StorageFilterMetrics) extends Serializable { + val metrics: StorageFilterMetrics, + val maxSplicedRowGroupBytes: Long) extends Serializable { // Codegen-produced predicates can be awkward to serialize from driver to executor, so we defer // construction to first use on the executor. @@ -81,27 +73,23 @@ class ParquetStorageFilter private ( def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow) /** - * Returns a new filter with the [[BoundReference]]s at `missingKeyLocalPositions` (positions in - * the local key-row layout, i.e. indices into [[keyColumnIndices]]) replaced by the constant the - * reader will actually materialize for that column, and the remaining [[BoundReference]]s - * renumbered to index into the reduced key-row layout. [[keyColumnIndices]] on the returned - * filter contains only the present columns in their original relative order. SQL metrics are - * shared with `this`. + * Returns a new filter for a physical file that is missing some key columns (schema evolution). + * The [[BoundReference]]s at `missingKeyLocalPositions`, which are indices into + * [[keyColumnIndices]], are replaced by `missingKeyValues`, and the remaining references are + * renumbered onto the reduced key-row layout. [[keyColumnIndices]] keeps the present columns in + * their original relative order, and SQL metrics are shared with `this`. * - * Used when a key column is missing from the physical parquet file (schema evolution). The - * predicate has to be evaluated against the substituted constant rather than skipped, because a - * null does not always mean `false` in a filter -- a `Coalesce`-wrapped reference still produces - * a non-null result, and `XxHash64` is `nullable = false` and hashes a null input to its seed. + * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing + * column: its existence DEFAULT when it has one, else null. `ParquetColumnVector` writes that + * default into the output vector, so substituting null instead would filter on a value the scan + * never returns and could drop matching rows. * - * `missingKeyValues(i)` is the internal-format value the reader produces for - * `missingKeyLocalPositions(i)`: the column's existence DEFAULT when it has one, else null. - * Passing the default matters for correctness -- `ParquetColumnVector` writes the existence - * default into the output vector for a missing column, so evaluating the predicate against null - * would filter on a value the scan never returns and could drop rows that match. + * The predicate has to be evaluated against the substitution rather than skipped, because a null + * key does not always mean `false`: a `Coalesce`-wrapped reference still produces a non-null + * result, and `XxHash64` is `nullable = false` and hashes a null input to its seed. * - * If all key positions are missing, the returned filter's [[boundExpression]] contains no - * [[BoundReference]]s and can be evaluated against [[InternalRow.empty]] to obtain a constant - * truth value (see [[evalAllMissing]]). + * With every key position missing, the result holds no [[BoundReference]] at all and + * [[evalAllMissing]] can read off its constant truth value. */ def rewriteForMissingKeys( missingKeyLocalPositions: Array[Int], @@ -117,7 +105,7 @@ class ParquetStorageFilter private ( case b: BoundReference => BoundReference(newPosOf(b.ordinal), b.dataType, b.nullable) } val newKeyColumnIndices = presentPositions.map(keyColumnIndices(_)).toArray - new ParquetStorageFilter(newKeyColumnIndices, rewritten, metrics) + new ParquetStorageFilter(newKeyColumnIndices, rewritten, metrics, maxSplicedRowGroupBytes) } /** @@ -151,7 +139,8 @@ object ParquetStorageFilter { def create( boundExpressions: Seq[Expression], requestedSchema: StructType, - metrics: StorageFilterMetrics = StorageFilterMetrics()): ParquetStorageFilter = { + metrics: StorageFilterMetrics = StorageFilterMetrics(), + maxSplicedRowGroupBytes: Long = Long.MaxValue): ParquetStorageFilter = { require(boundExpressions.nonEmpty, "storage filters must be non-empty; callers with nothing to push must not call create") val expr = boundExpressions.reduce(And) @@ -159,14 +148,10 @@ object ParquetStorageFilter { // The requested-schema ordinals this predicate reads, deduplicated (a column referenced twice // is still one key column) and sorted. // - // `sorted` is load-bearing, not cosmetic. It is not needed to keep `keyColumnIndices` and the - // remapped BoundReferences consistent with each other -- both derive from this list, so any - // order would agree. It is needed by a third consumer that does NOT go through the remapping: - // `VectorizedParquetRecordReader.nextBatchSplicing` walks the output batch slots in ascending - // order and pulls the survivor queues in key-row-position order, so it pairs "the k-th smallest - // key slot" with "key-row position k". That pairing is the identity only while this list is - // ascending. It cannot recover the order itself, because the reader records which slots are - // keys in a boolean array (`isKeyTopLevel`) that does not preserve their position here. + // `sorted` is load-bearing. Both `keyColumnIndices` and the remapped references derive from + // this list, so any order would keep those two consistent, but the reader's emit path does + // not go through the remapping: it pairs the k-th key slot of the output batch with key-row + // position k, which is the identity only while this list is ascending. val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted require(originalOrdinals.nonEmpty, s"storage filter $expr has no bound reference to a key column") @@ -185,18 +170,16 @@ object ParquetStorageFilter { case b: BoundReference => BoundReference(indexMap(b.ordinal), b.dataType, b.nullable) } - new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics) + new ParquetStorageFilter(originalOrdinals.toArray, remapped, metrics, maxSplicedRowGroupBytes) } /** - * Whether `dt` is usable as a storage-filter key column type. - * - * This is the single authority on key-type eligibility: - * `FileSourceStrategy.extractStorageFilters` consults it at planning time and [[create]] - * re-checks it, so the vectorized reader's per-type value copier - * (`VectorizedParquetRecordReader.copierFor`) is only ever asked for a type listed here. The two - * must stay in lockstep -- adding a type here without teaching `copierFor` about it turns a - * planning-time rejection into a task failure. + * Whether `dt` is usable as a storage-filter key column type. This is the single authority on + * that: [[isSupportedStorageFilter]] consults it for what the planner asks, and [[create]] + * re-checks it, so the reader's per-type value copier + * (`VectorizedParquetRecordReader.copierFor`) is only ever asked for a type listed here. Adding + * a type here without teaching `copierFor` about it turns a planning-time rejection into a task + * failure. * * Narrower than `AtomicType`, for two different reasons: * - `VariantType` cannot be supported: its Parquet representation is a group, not a primitive @@ -217,4 +200,18 @@ object ParquetStorageFilter { case _: StringType | _: BinaryType => true case _ => false } + + /** + * Whether the reader can evaluate `expr` as a storage filter, which is what + * `ParquetFileFormat.supportsStorageFilter` answers for the planner. + */ + def isSupportedStorageFilter(expr: Expression): Boolean = expr match { + case bloom: BloomFilterMightContain => + // The whole conjunct has to be the bloom, not something with a bloom nested under an OR or a + // NOT: the reader evaluates the expression it is given and treats a false as "drop this row". + // Every reference is checked, not just the ones on the value side, because [[create]] binds + // and type-checks all of them. + bloom.references.forall(a => isSupportedKeyType(a.dataType)) + case _ => false + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala index b0026bdbd7083..5243b728988a9 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala @@ -33,16 +33,17 @@ import org.apache.parquet.hadoop.{ParquetInputFormat, ParquetOutputFormat} import org.apache.spark.paths.SparkPath import org.apache.spark.sql.{sources, QueryTest, Row, SparkSession} import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{And, Attribute, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64} +import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64} import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter} import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec} -import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile} +import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile, UnsupportedFileReadException} import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} import org.apache.spark.sql.functions.col import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types._ import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} +import org.apache.spark.util.Utils import org.apache.spark.util.sketch.BloomFilter /** @@ -64,8 +65,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { .repartition(1) .write .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) - // Dictionary encoding off keeps row-group sizing predictable. Note this does NOT disable the - // column index, despite what an earlier version of this comment claimed. + // Dictionary encoding off keeps row-group sizing predictable. The column index is still + // written either way. .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") // A small page size gives each row group several pages per column, which is what lets // column-index filtering produce a row range narrower than the whole row group. @@ -77,26 +78,36 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } // Collects all rows from a reader initialized with the given storage filter. + // + // `tryInitializeResource` closes the reader if anything inside throws and leaves it open + // otherwise, which is the contract these helpers need: the caller closes it once its assertions + // pass. Without it a failure in the read loop -- what these tests are looking for -- would leak + // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade + // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at + // construction. private def readAll( filePath: String, storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = { - val reader = new VectorizedParquetRecordReader(false, 4096) - reader.setStorageFilter(storageFilter) - reader.initialize(filePath, java.util.Arrays.asList("k", "v")) - reader.initBatch(new StructType(), null) - val collected = mutable.ArrayBuffer[(Long, String)]() - while (reader.nextBatch()) { - val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] - val n = batch.numRows() - val kVec = batch.column(0) - val vVec = batch.column(1) - var i = 0 - while (i < n) { - collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString)) - i += 1 + Utils.tryInitializeResource { + new VectorizedParquetRecordReader(false, 4096) + } { reader => + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, java.util.Arrays.asList("k", "v")) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[(Long, String)]() + while (reader.nextBatch()) { + val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] + val n = batch.numRows() + val kVec = batch.column(0) + val vVec = batch.column(1) + var i = 0 + while (i < n) { + collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString)) + i += 1 + } } + (collected.toSeq, reader) } - (collected.toSeq, reader) } // Builds a `k >= threshold` storage filter bound to position 0. @@ -134,22 +145,25 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { storageFilter: ParquetStorageFilter, extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T, capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) = { - val reader = new VectorizedParquetRecordReader(false, capacity) - reader.setStorageFilter(storageFilter) - reader.initialize(filePath, java.util.Arrays.asList("k")) - reader.initBatch(new StructType(), null) - val collected = mutable.ArrayBuffer[T]() - while (reader.nextBatch()) { - val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] - val n = batch.numRows() - val kVec = batch.column(0) - var i = 0 - while (i < n) { - collected += extract(kVec, i) - i += 1 + Utils.tryInitializeResource { + new VectorizedParquetRecordReader(false, capacity) + } { reader => + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, java.util.Arrays.asList("k")) + reader.initBatch(new StructType(), null) + val collected = mutable.ArrayBuffer[T]() + while (reader.nextBatch()) { + val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] + val n = batch.numRows() + val kVec = batch.column(0) + var i = 0 + while (i < n) { + collected += extract(kVec, i) + i += 1 + } } + (collected.toSeq, reader) } - (collected.toSeq, reader) } // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the @@ -250,7 +264,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("key-only projection: phase 2 is skipped and both byte-avoided metrics are zero") { // When the projected schema contains only the bloom key, phase 2 is skipped entirely - // (`nonKeyRequestedSchema == null` in the reader). All output rows come from the per-key-column + // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero: // there are no non-key bytes to skip. @@ -640,6 +654,32 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } + test("FileSourceStrategy leaves the bloom behind when every projected column is a key column") { + // Nothing is left for phase 2 to prune: the reader would read the same column for the same rows + // as a plain scan, since it has to read a key column to evaluate the filter on it, and would + // add only the cost of evaluating the predicate outside the generated code. + withBloomFilterTables { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val df = spark.sql("SELECT bf1.k FROM bf1 JOIN bf2 ON bf1.k = bf2.k WHERE bf2.v = 5") + val plan = df.queryExecution.executedPlan + val storageBlooms = countBloomFiltersInStorageFilters(plan) + val postScanBlooms = countBloomFiltersInPostScanFilters(plan) + assert(storageBlooms == 0, + s"expected no bloom on scan.storageFilters for an all-keys projection; got " + + s"$storageBlooms.\nPlan:\n$plan") + assert(postScanBlooms >= 1, + s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" + + s"Plan:\n$plan") + assert(df.collect().map(_.getLong(0)).toSet == Set(5L), + "and the query must still return the joined key") + } + } + } + test("FileSourceStrategy leaves bloom filter as post-scan FilterExec when conf is off") { withBloomFilterTables { withSQLConf( @@ -659,6 +699,38 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } + test("ignoreCorruptFiles does not swallow a failure to honor a storage filter") { + // The reader has to fail when a file cannot support late materialization, because the planner + // removed the conjunct from the post-scan Filter. FileScanRDD and FilePartitionReader would + // skip the rest of such a file under ignoreCorruptFiles, silently dropping readable rows, so + // the exception the reader throws is excluded from that. + assert(!DataSourceUtils.shouldIgnoreCorruptFileException( + new UnsupportedFileReadException("cannot honor a storage filter"))) + // The generic reader failures it is carved out of stay swallowed. + assert(DataSourceUtils.shouldIgnoreCorruptFileException(new IllegalStateException("corrupt"))) + assert(DataSourceUtils.shouldIgnoreCorruptFileException(new java.io.IOException("truncated"))) + } + + test("the feature still engages when ignoreCorruptFiles is on") { + withBloomFilterTables { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.IGNORE_CORRUPT_FILES.key -> "true", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val (plan, rows) = runBloomFilterJoin() + val storageBlooms = countBloomFiltersInStorageFilters(plan) + assert(storageBlooms == 1, + s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan") + assert(countBloomFiltersInPostScanFilters(plan) == 0, + s"expected no bloom left above the scan.\nPlan:\n$plan") + assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)), + s"and the query must still return the joined row; got ${rows.mkString(", ")}") + } + } + } + test("FileSourceStrategy extraction preserves query results") { withBloomFilterTables { val baseConf = Map( @@ -678,6 +750,70 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } + Seq(Seq("k", "v"), Seq("k")).foreach { projection => + test("a row group over the splice cap is read the plain way, same rows, more bytes " + + s"(projection ${projection.mkString(",")})") { + // Past the cap a row group is read the plain way: phase 2 takes every projected column, key + // columns included, so nothing is buffered. Two things have to hold. The rows must not + // change, since a fallback that quietly dropped the predicate would return extra rows. And + // the fallback must actually have happened, which is observable: it reads the key column a + // second time, so it transfers strictly more bytes. Without that second assertion the test + // would pass even if the cap never reached the reader. + // + // The batch size is what makes the cap reachable at all: the reader examines the count only + // once a batch worth of survivors per key column has been buffered, so 51 survivors have to + // cross that line more than once. + // + // The `k` projection is the interesting one: an all-keys projection normally skips phase 2 + // entirely, so past the cap it has to read the key column there like any other column. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + val fileSchema = StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))) + val readSchema = StructType(projection.map(fileSchema(_))) + val storageFilters = + Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))) + + def run(maxSplicedBytes: String): (Int, Long) = { + withSQLConf( + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES.key -> + maxSplicedBytes, + SQLConf.PARQUET_VECTORIZED_READER_BATCH_SIZE.key -> "16") { + val hadoopConf = spark.sessionState.newHadoopConf() + hadoopConf.set(s"fs.${CountingLocalFileSystem.scheme}.impl", + classOf[CountingLocalFileSystem].getName) + hadoopConf.setBoolean(s"fs.${CountingLocalFileSystem.scheme}.impl.disable.cache", true) + val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters( + spark, fileSchema, new StructType(), readSchema, Nil, storageFilters, + Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty) + val file = PartitionedFile( + InternalRow.empty, + SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"), + 0, + new File(path).length()) + CountingLocalFileSystem.reset() + val emitted = readerFn(file).asInstanceOf[Iterator[Object]].map { + case batch: ColumnarBatch => batch.numRows() + case _ => 1 + }.sum + (emitted, CountingLocalFileSystem.bytesRead()) + } + } + + val (splicedRows, splicedBytes) = run("64MB") + val (plainRows, plainBytes) = run("1b") + assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows") + assert(plainRows == splicedRows, + s"rows differ past the cap: plain=$plainRows spliced=$splicedRows") + assert(plainBytes > splicedBytes, + s"past the cap the key column is read twice, so the read must be larger; " + + s"plain=$plainBytes spliced=$splicedBytes") + } + } + } + test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") { // `ParquetStorageFilter.test` evaluates the predicate without calling // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a @@ -754,21 +890,24 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { useOffHeap: Boolean = false, partitionColumns: StructType = new StructType(), partitionValues: InternalRow = null): (Seq[T], VectorizedParquetRecordReader) = { - val reader = new VectorizedParquetRecordReader(useOffHeap, capacity) - reader.setStorageFilter(storageFilter) - reader.initialize(filePath, columns.asJava) - reader.initBatch(partitionColumns, partitionValues) - val collected = mutable.ArrayBuffer[T]() - while (reader.nextBatch()) { - val batch = reader.resultBatch() - var i = 0 - val n = batch.numRows() - while (i < n) { - collected += extract(batch, i) - i += 1 + Utils.tryInitializeResource { + new VectorizedParquetRecordReader(useOffHeap, capacity) + } { reader => + reader.setStorageFilter(storageFilter) + reader.initialize(filePath, columns.asJava) + reader.initBatch(partitionColumns, partitionValues) + val collected = mutable.ArrayBuffer[T]() + while (reader.nextBatch()) { + val batch = reader.resultBatch() + var i = 0 + val n = batch.numRows() + while (i < n) { + collected += extract(batch, i) + i += 1 + } } + (collected.toSeq, reader) } - (collected.toSeq, reader) } // Renders one column value as a string using its *internal* representation, so the splicing path @@ -844,11 +983,10 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { ("binary", "CAST(LPAD(CAST(id AS STRING), 5, '0') AS BINARY)", BinaryType, "00090".getBytes("UTF-8"))) - // Run every key type both without and WITH dictionary encoding. Dictionary encoding is parquet's - // production default, and it is the case where the phase 1 scratch vectors carry a Dictionary - // plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's decode branch - // rather - // than straight out of the value array. + // Run every key type both without and WITH dictionary encoding. Dictionary encoding is + // parquet's production default, and it is the case where the phase 1 scratch vectors carry a + // Dictionary plus dictionaryIds, so each ValueCopier reads through WritableColumnVector's + // decode branch rather than straight out of the value array. for { (name, keyExpr, dt, threshold) <- keyTypeCases dictionary <- Seq(false, true) @@ -1301,8 +1439,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("complex non-key column is assembled correctly under splicing") { // Phase 2 reads non-key columns through initColumnReader's recursion and cv.assemble(); a - // struct - // column exercises both, which a flat projection never does. + // struct column exercises both, which a flat projection never does. withTempDir { dir => val outDir = new File(dir, "structs").getAbsolutePath spark.range(1, 201) @@ -1332,6 +1469,30 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // ----- Planner gates and the lost-filter invariant ----- + test("supportsStorageFilter is what decides, and a subclass answers false") { + // The planner asks the format rather than testing its class, so the expression shapes and the + // column types a reader can evaluate stay in its own package. A ParquetFileFormat subclass + // still answers false: it may customize reading by overriding buildReaderWithPartitionValues, + // and a scan with storage filters routes through buildReaderWithStorageFilters instead, which + // would bypass whatever the subclass does. + val format = new ParquetFileFormat() + val subclass = new ParquetFileFormat() {} + val bloom = BloomFilterMightContain( + Literal.create(null, BinaryType), + XxHash64(Seq(AttributeReference("k", LongType)()), 42L)) + assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported") + assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support") + // Not a bloom at all, and a bloom on a type the value copier has no branch for. + assert(!format.supportsStorageFilter(Literal.TrueLiteral)) + val onVariant = BloomFilterMightContain( + Literal.create(null, BinaryType), + XxHash64(Seq(AttributeReference("v", VariantType)()), 42L)) + assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf") + // And the default is no support at all. + assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom)) + } + + test("bloom stays in the post-scan Filter when the vectorized reader is unavailable") { // The whole lost-filter safety argument rests on this gate: if the reader cannot do late // materialization, the planner must NOT move the bloom out of the post-scan Filter. @@ -1444,9 +1605,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // The planner's extraction gate is `fileFormat.supportBatch`, which does not look at // whole-stage codegen, while `FileSourceScanExec.supportsColumnar` does. So with codegen off // the bloom is still extracted, `returningBatch` is false, and the reader serves the spliced - // batch - // one row at a time. That combination is the only one where the planner's gate is weaker than - // the runtime's, and only the reader-level test covered the row-at-a-time path. + // batch one row at a time. That is the only combination where the planner's gate is weaker + // than the runtime's. withBloomFilterTables { val baseConf = Map( SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", @@ -1475,8 +1635,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("off-heap column vectors through the planner: spliced values survive the free") { // Off-heap is where the vector lifecycle actually bites: the previous batch's key vectors are // freed at the next nextBatch(), so a stale reference reads released native memory rather than - // old bytes. Only the reader-level tests passed useOffHeap = true; this drives it through the - // planner, where the batch also crosses ColumnarToRowExec. + // old bytes. This drives it through the planner, where the batch also crosses + // ColumnarToRowExec. withTempDir { dir => val rows = (1L to 200L).map(i => (i, s"v_$i")) val path = writeParquetFile(dir, rows, rowGroupSize = 256L) @@ -1854,8 +2014,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("one of two key columns missing from a file: splicing runs with the rewritten predicate") { // The most intricate branch of initializeLateMaterialization: splicing engages with a predicate // that has one Literal substituted and one BoundReference renumbered, the missing key's field - // lands in nonKeyRequestedSchema, and its output slot is filled by ParquetColumnVector. Only - // the all-missing case was covered end to end before. + // lands among the non-key columns, and its output slot is filled by ParquetColumnVector. // // The SELECT order (a, b, c) also differs from the table's (a, c, b), so this covers a // projection whose order does not match the relation's dataSchema. From c255f8892b9cb9edeee133a0e3828e1552a3ad3d Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Thu, 24 Sep 2026 15:55:08 +0200 Subject: [PATCH 3/6] Review fixes (round 2) - Keep the pushed conjunct in the post-scan Filter, so the scan's filtering is a copy rather than a transfer of responsibility - Cap the heap the phase-2 row ranges and the spliced key buffer take together, and give the filter up for a row group that would not fit - Retry a row group without late materialization when the file has no offset index - Push an expression only when it is safe to evaluate on every row of a row group - Cache the prepared filter's rewrites for files missing a key column - Remove UnsupportedFileReadException - Fold the splicing emit path into nextBatch and the batch release into one close body, and put every write and read helper of the suite on one - Measure a whole-file skip against the rows the pushed data filter kept, the baseline every other skip path uses, and skip the byte baseline a row group that already gave the filter up cannot report - Make the dictionary-encoded key-type tests write dictionary-encoded files, and assert the encoding they got - Cover what nothing asserted: key ordinals collected out of order, row groups that splice and row groups over the cap in both orders, the cap reached inside phase 1's survivor loop, the second key read in the byte metric, the rows a whole-file skip reports, and the row identity on the all-keys path - Correct the comments the copy model, the folded emit path and three wrong justifications left behind --- .../apache/spark/sql/internal/SQLConf.scala | 91 +-- .../datasources/parquet/ParquetReadState.java | 7 + .../VectorizedParquetRecordReader.java | 461 ++++++++------ .../sql/execution/DataSourceScanExec.scala | 12 +- .../datasources/DataSourceUtils.scala | 8 - .../execution/datasources/FileFormat.scala | 32 +- .../datasources/FileSourceStrategy.scala | 58 +- .../UnsupportedFileReadException.scala | 35 -- .../parquet/ParquetFileFormat.scala | 24 +- .../parquet/ParquetStorageFilter.scala | 70 ++- .../parquet/ParquetStorageFilterSuite.scala | 571 ++++++++++++------ 11 files changed, 823 insertions(+), 546 deletions(-) delete mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 3d6bf4b198992..a0d64ff65b5b2 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1885,42 +1885,6 @@ object SQLConf { .booleanConf .createWithDefault(true) - val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED = - buildConf("spark.sql.parquet.storageFilterPushdown.enabled") - .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " + - "(e.g. bloom filters from join runtime filtering) at the scan level using late " + - "materialization: read key columns first, evaluate the filter per row, then read data " + - "columns restricted to surviving rows. This is a planning-time decision only: when " + - "false, no storage filter is attached to a scan in the first place and the filter is " + - "applied as an ordinary post-scan filter instead. Note that the surviving key values of " + - "a whole row group are buffered before the first batch of that row group is produced, " + - "so a task holds up to one extra copy of the key columns for one row group. Note also " + - "that reading only the surviving rows of a row group needs a Parquet offset index. A " + - "scan of a file written without a page index fails as soon as its filter rejects part " + - "of a row group, so set this to false to read such a file.") - .version("5.0.0") - .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) - .booleanConf - .createWithDefault(false) - - val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES = - buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes") - .internal() - .doc("Largest key-column buffer, in bytes, that the vectorized Parquet reader will hold to " + - "splice surviving key values into its output batches. Splicing buffers one key value per " + - "surviving row of a row group, so the reader counts what it has buffered and gives that " + - "row group up once the count passes this, reading every projected column of the " + - "surviving rows in one go instead, which costs one extra read of the key columns. What " + - "is counted is the buffered values and their per-row overhead, not the backing arrays, " + - "which a column vector may grow beyond that. The count is examined whenever a batch " + - "worth of survivors per key column has been buffered, so a row group whose survivors fit " + - "in a single batch is never given up: it holds no more than the plain read path does.") - .version("5.0.0") - .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) - .bytesConf(ByteUnit.BYTE) - .checkValue(_ > 0, "must be positive") - .createWithDefaultString("64MB") - val PARQUET_FILTER_PUSHDOWN_DATE_ENABLED = buildConf("spark.sql.parquet.filterPushdown.date") .doc("If true, enables Parquet filter push-down optimization for Date. " + s"This configuration only has an effect when '${PARQUET_FILTER_PUSHDOWN_ENABLED.key}' is " + @@ -1983,6 +1947,49 @@ object SQLConf { .checkValue(threshold => threshold >= 0, "The threshold must not be negative.") .createWithDefault(10) + val PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED = + buildConf("spark.sql.parquet.storageFilterPushdown.enabled") + .doc("If true, allows the vectorized Parquet reader to evaluate runtime storage filters " + + "(e.g. bloom filters from join runtime filtering) at the scan level using late " + + "materialization: read key columns first, evaluate the filter per row, then read data " + + "columns restricted to surviving rows. This is a planning-time decision only: when " + + "false, no storage filter is attached to a scan in the first place and the filter is " + + "applied as an ordinary post-scan filter alone. A pushed filter stays in the post-scan " + + "filter as well, the way a pushed data filter does, so honoring it is optional: a reader " + + "that meets a file it cannot prune, one written without a Parquet page index say, reads " + + "it the way a plain scan would. That costs the filter's own evaluation, plus one " + + "key-column read of the first row group the reader tries it on, since a missing page " + + "index is only reported by attempting the read. Note that the surviving key values of a " + + "whole row group are buffered before the " + + "first batch of that row group is produced, so a task holds up to one extra copy of the " + + "key columns for one row group.") + .version("5.0.0") + .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) + .booleanConf + .createWithDefault(false) + + val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES = + buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes") + .internal() + .doc("Largest key-column buffer, in bytes, that the vectorized Parquet reader will hold to " + + "splice surviving key values into its output batches. Splicing buffers one key value per " + + "surviving row of a row group, so the reader counts what it has buffered and gives that " + + "row group up once the count passes this, reading every projected column of the " + + "surviving rows in one go instead, which costs one extra read of the key columns. What " + + "is counted is the buffered values and their per-row overhead, not the backing arrays, " + + "which a column vector may grow beyond that. The count is examined whenever a batch " + + "worth of survivors per key column has been buffered, so a row group whose survivors fit " + + "in a single batch is never weighed at all: it holds no more than the plain read path " + + "does. The same limit covers the row ranges the surviving rows fall into, which every " + + "column reader of the second phase holds a copy of, so the two share one budget. A row " + + "group whose ranges alone pass it is read without the filter applied at all, which is " + + "correct but as slow as not pushing the filter.") + .version("5.0.0") + .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) + .bytesConf(ByteUnit.BYTE) + .checkValue(_ > 0, "must be positive") + .createWithDefaultString("64MB") + val PARQUET_AGGREGATE_PUSHDOWN_ENABLED = buildConf("spark.sql.parquet.aggregatePushdown") .doc("If true, aggregates will be pushed down to Parquet for optimization. Support MIN, MAX " + "and COUNT as aggregate expression. For MIN/MAX, support boolean, integer, float and date " + @@ -9142,12 +9149,6 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def parquetFilterPushDown: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_ENABLED) - def parquetStorageFilterPushdownEnabled: Boolean = - getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED) - - def parquetStorageFilterPushdownMaxSplicedRowGroupBytes: Long = - getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES) - def parquetFilterPushDownDate: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_DATE_ENABLED) def parquetFilterPushDownTimestamp: Boolean = getConf(PARQUET_FILTER_PUSHDOWN_TIMESTAMP_ENABLED) @@ -9160,6 +9161,12 @@ class SQLConf extends Serializable with Logging with SqlApiConf { def parquetFilterPushDownInFilterThreshold: Int = getConf(PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD) + def parquetStorageFilterPushdownEnabled: Boolean = + getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED) + + def parquetStorageFilterPushdownMaxSplicedRowGroupBytes: Long = + getConf(PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES) + def parquetAggregatePushDown: Boolean = getConf(PARQUET_AGGREGATE_PUSHDOWN_ENABLED) def orcFilterPushDown: Boolean = getConf(ORC_FILTER_PUSHDOWN_ENABLED) diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java index 7a47d350af616..e29a16a834397 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java @@ -179,4 +179,11 @@ void nextRange() { */ private record RowRange(long start, long end) { } + + /** + * What one {@link RowRange} costs on the heap, for a caller that has to budget for the list this + * class builds: two longs, their object header, and the slot in the list holding them. It lives + * here because {@link RowRange} is private, so a caller cannot measure it. + */ + static final int ESTIMATED_ROW_RANGE_BYTES = 40; } diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index e178f3c12ea29..059f8beb27502 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -54,10 +54,13 @@ import org.apache.parquet.schema.Types; import org.apache.spark.SparkUnsupportedOperationException; +import org.apache.spark.internal.LogKeys; +import org.apache.spark.internal.MDC; +import org.apache.spark.internal.SparkLogger; +import org.apache.spark.internal.SparkLoggerFactory; import org.apache.spark.memory.MemoryMode; import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns; import org.apache.spark.sql.catalyst.InternalRow; -import org.apache.spark.sql.execution.datasources.UnsupportedFileReadException; import org.apache.spark.sql.execution.metric.SQLMetric; import org.apache.spark.sql.execution.vectorized.ColumnVectorUtils; import org.apache.spark.sql.execution.vectorized.ConstantColumnVector; @@ -82,6 +85,9 @@ */ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBase { + private static final SparkLogger LOG = + SparkLoggerFactory.getLogger(VectorizedParquetRecordReader.class); + // The capacity of vectorized batch. private int capacity; @@ -206,11 +212,21 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa private boolean[] keyVariableLength; /** Key-value bytes buffered for the row group currently loading. */ private long splicedBytes; + /** Ranges the surviving rows of the row group currently loading fall into. */ + private long survivorRangeCount; + /** Whether the row group currently loading is read without the filter applied at all. */ + private boolean filterGivenUp; + /** + * Whether this file has no Parquet offset index for some projected column, which parquet only + * reports by throwing when a read asks for part of a block. It is a property of the file, so it + * is learned once: later row groups skip phase 1 rather than evaluate a filter they cannot use. + */ + private boolean fileHasNoOffsetIndex; /** - * Whether the row group currently loaded is spliced. Starts true for every row group and can turn - * false in phase 1, once the survivors buffered pass the cap; false means phase 2 read every - * projected column, key columns included, so the emit path takes them straight from the - * persistent batch. + * Whether the row group currently loaded is spliced. It starts true unless the file is already + * known to have no offset index, and turns false in phase 1 once the survivors buffered pass the + * cap. False means phase 2 read every projected column, key columns included, so the emit path + * takes them straight from the persistent batch. */ private boolean spliceCurrentRowGroup; private int nextBlockIndex; @@ -230,11 +246,9 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit * path splices them back into the output batch, so phase 2 never reads the key columns. * - *

The alternative is for phase 2 to read the key columns again under the surviving row - * ranges, which needs none of this state but pays a second read of them. That second read is - * the whole cost of the scan when the projection is key columns alone, the shape a pushed join - * filter most often produces: splicing skips phase 2 there, while a re-read has nothing left to - * skip. + *

The alternative is for phase 2 to read the key columns again under the surviving row ranges, + * which needs none of this state but pays a second read of them for every row group. Nothing + * absorbs that read on object storage, where it is a new GET rather than a page-cache hit. * *

{@link #isKeyTopLevel} marks the top-level slots that emit takes from the queues rather * than from a phase-2 read. {@link #keyVectorQueues} holds one queue per present key column of @@ -340,27 +354,22 @@ public void close() throws IOException { // stream, so they are chained through finally blocks: one failing vector close must not leak // the rest. try { - if (isKeyTopLevel != null) { - // Splicing: the emitted batch is a view whose slots alias persistentBatchColumns (non-key - // and partition) and the head of each survivor queue (key slots). We never call - // columnarBatch.close() because that would re-close those shared vectors after the direct - // closes below, freeing the same buffer twice. - try { - if (persistentBatchColumns != null) { - for (ColumnVector v : persistentBatchColumns) { - if (v != null) v.close(); - } - persistentBatchColumns = null; + // The batch's vectors are closed through `persistentBatchColumns` rather than through + // `columnarBatch.close()`, which lets both paths share one body. While splicing the emitted + // batch is a view whose slots alias these vectors (non-key and partition) and the head of + // each survivor queue (key slots), so closing it would re-close a shared vector and free the + // same buffer twice. Without splicing its array is this one. + try { + if (persistentBatchColumns != null) { + for (ColumnVector v : persistentBatchColumns) { + if (v != null) v.close(); } - spliceBatchColumns = null; - columnarBatch = null; - } finally { - closeSplicingState(); + persistentBatchColumns = null; } - } else if (columnarBatch != null) { - columnarBatch.close(); + spliceBatchColumns = null; columnarBatch = null; - persistentBatchColumns = null; + } finally { + closeSplicingState(); } } finally { try { @@ -432,14 +441,13 @@ private void initBatch( constantColumnLength = partitionColumns.fields().length; } - // Every slot gets a vector, key columns included. While splicing, a key slot's vector is unused - // -- the emitted batch takes that slot from the survivor queues -- but a row group read the - // plain way past the buffer cap reads into it, and one capacity-sized vector per key column is + // Every slot gets a vector, key columns included. While splicing a key slot's vector is unused, + // since the emitted batch takes that slot from the survivor queues. A row group read the plain + // way past the buffer cap does read into it, and one capacity-sized vector per key column is // cheap next to the buffer the cap is there to bound. ColumnVector[] vectors = allocateColumns( capacity, batchSchema, memMode == MemoryMode.OFF_HEAP, constantColumnLength); - columnarBatch = new ColumnarBatch(vectors); persistentBatchColumns = vectors; if (isKeyTopLevel != null) { // Splicing hands out one batch for the whole read, over its own array, whose key slots the @@ -447,6 +455,8 @@ private void initBatch( // included, so rewriting a slot is what publishes it. spliceBatchColumns = vectors.clone(); columnarBatch = new ColumnarBatch(spliceBatchColumns); + } else { + columnarBatch = new ColumnarBatch(vectors); } columnVectors = new ParquetColumnVector[sparkSchema.fields().length]; @@ -555,10 +565,12 @@ public void enableReturningBatches() { * Advances to the next batch of rows. Returns false if there are no more. */ public boolean nextBatch() throws IOException { - if (isKeyTopLevel != null) return nextBatchSplicing(); + releasePublishedKeyVectors(); for (ParquetColumnVector vector : columnVectors) { vector.reset(); } + // Zeroed before the terminal checks below, so a terminal call cannot leave a spliced batch + // pointing at the key vectors just released, which off heap is freed memory. columnarBatch.setNumRows(0); if (hitEndOfData) return false; if (rowsReturned >= totalRowCount) return false; @@ -566,8 +578,20 @@ public boolean nextBatch() throws IOException { if (hitEndOfData) return false; int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); - readPersistentColumns(num, /* skipKeySlots= */ false); - // If needed, compute row indexes within a file. + // A spliced row group takes its key slots from the survivor queues, so phase 2 skipped them. + // Everything else read every projected column: a plain read with no storage filter at all, or a + // row group that gave splicing up, whose batch needs its key slots put back to the persistent + // vectors a previous row group rewrote. `spliceCurrentRowGroup` stays false without a storage + // filter, so it answers for all three. + if (spliceCurrentRowGroup) { + publishSurvivorKeyVectors(num); + } else if (spliceBatchColumns != null) { + System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length); + } + readPersistentColumns(num, /* skipKeySlots= */ spliceCurrentRowGroup); + // If needed, compute row indexes within a file. The row-index column is identified by name + // (ROW_INDEX_TEMPORARY_COLUMN_NAME), a synthetic metadata column no storage filter references, + // so its slot is always a non-key one and its ParquetColumnVector is always the persistent one. if (rowIndexGenerator != null) { rowIndexGenerator.populateRowIndex(columnVectors, num); } @@ -576,8 +600,8 @@ public boolean nextBatch() throws IOException { } /** - * Reads {@code num} rows into the persistent batch slots and assembles them. All three emit paths - * share this: the plain read, a spliced row group (which skips the key slots, since the emitted + * Reads {@code num} rows into the persistent batch slots and assembles them. Every case goes + * through here: the plain read, a spliced row group (which skips the key slots, since the emitted * batch takes those from the survivor queues), and a row group read the plain way past the buffer * cap (which reads every slot). * @@ -609,76 +633,32 @@ private void finishBatch(int num) { } /** - * Splicing emit path. Publishes one survivor key vector per key column into the batch, reads the - * non-key slots for {@code num} rows, and hands out the same {@link ColumnarBatch} every time, - * its key slots rewritten in place. The batch is a view over vectors owned elsewhere; see - * {@link #close()} and {@link #closeSplicingState()}. + * Points the batch's key slots at one survivor key vector each, which is what publishes them: the + * same {@link ColumnarBatch} is handed out every time, over an array it holds by reference. The + * queues keep owning those vectors until the next batch releases them, so a phase-2 read that + * throws afterwards leaves them reachable for {@link #close()}. */ - private boolean nextBatchSplicing() throws IOException { - releasePublishedKeyVectors(); - for (ParquetColumnVector cv : columnVectors) { - cv.reset(); - } - // Zero the outgoing batch before the terminal checks below, as the plain path does. Otherwise a - // terminal call leaves the batch pointing at key vectors that were just closed -- off-heap, - // that is freed memory. - if (columnarBatch != null) columnarBatch.setNumRows(0); - if (hitEndOfData) return false; - if (rowsReturned >= totalRowCount) return false; - checkEndOfRowGroup(); - if (hitEndOfData) return false; - - int num = (int) Math.min(capacity, totalCountLoadedSoFar - rowsReturned); - - if (!spliceCurrentRowGroup) { - // This row group was read the plain way: phase 2 took every projected column, so every slot - // of the batch comes from the persistent one. - readPersistentColumns(num, /* skipKeySlots= */ false); - if (rowIndexGenerator != null) { - rowIndexGenerator.populateRowIndex(columnVectors, num); - } - System.arraycopy(persistentBatchColumns, 0, spliceBatchColumns, 0, spliceBatchColumns.length); - finishBatch(num); - return true; - } - + private void publishSurvivorKeyVectors(int num) { for (int i = 0; i < keyVectorQueues.length; i++) { if (keyVectorQueues[i].isEmpty()) { // Unreachable: the queues hold exactly the survivors phase 1 accumulated, and the emit loop - // is driven by that same count. Named rather than left to NoSuchElementException because - // this one must not be swallowed: under ignoreCorruptFiles that would silently drop the - // rest of a healthy file's rows. - throw new UnsupportedFileReadException(String.format( + // is driven by that same count. Named rather than left to NoSuchElementException. + throw new IllegalStateException(String.format( "Storage-filter survivor queue %d of row group %d in %s ran out with %d rows still to " + "emit", i, nextBlockIndex - 1, lateMatReader.getFile(), num)); } } - // The queues keep owning these until the next emit releases them, so a phase-2 read below that - // throws leaves them reachable for `close()`. keyVectorsPublished = true; // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap - // key columns in the output batch. + // key columns in the output batch. Only key slots are touched, since a non-key slot never + // holds anything but its persistent vector. int keyIdx = 0; - for (int i = 0; i < spliceBatchColumns.length; i++) { - if (i < isKeyTopLevel.length && isKeyTopLevel[i]) { - spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst(); - } else { - spliceBatchColumns[i] = persistentBatchColumns[i]; - } + for (int i = 0; i < isKeyTopLevel.length; i++) { + if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst(); } - - readPersistentColumns(num, /* skipKeySlots= */ true); - if (rowIndexGenerator != null) { - // Row-index column is identified by name (ROW_INDEX_TEMPORARY_COLUMN_NAME), which is a - // synthetic metadata column never referenced by a storage filter, so its slot is a non-key - // slot with a persistent ParquetColumnVector. - rowIndexGenerator.populateRowIndex(columnVectors, num); - } - finishBatch(num); - return true; } /** @@ -710,13 +690,13 @@ private void initializeInternal() throws IOException, UnsupportedOperationExcept * *

It does not engage in one real case: every key column is missing from this physical file * under schema evolution. The predicate is then rewritten with each missing key replaced by the - * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once - * -- true keeps the file unfiltered, false or null skips it. + * constant the reader materializes for it (its existence DEFAULT, else null) and evaluated once. + * True keeps the file unfiltered, false or null skips it. * - *

Every other precondition is guaranteed by - * {@code FileSourceStrategy.extractStorageFilters} and {@code ParquetStorageFilter.create}, and a - * violation throws rather than falling back: extraction has already removed the filter from the - * post-scan Filter, so not applying it would return rows the query rejected. + *

Everything else the filter needs is guaranteed by + * {@code FileSourceStrategy.storageFiltersFor} and {@code ParquetStorageFilter.create}, so a + * violation of it here is a planner bug and is asserted rather than handled. A file the reader + * simply cannot prune is a different matter: it reads it the way a plain scan would. */ public void setStorageFilter(ParquetStorageFilter storageFilter) { this.storageFilter = storageFilter; @@ -725,17 +705,14 @@ public void setStorageFilter(ParquetStorageFilter storageFilter) { private void initializeLateMaterialization() throws IOException { lateMatReader = reader.getUnderlyingReader(); if (lateMatReader == null) { - // Unreachable: the only ParquetRowGroupReader ParquetFileFormat builds is - // ParquetRowGroupReaderImpl, which exposes its reader. - throw new UnsupportedFileReadException( - "Storage-filter pushdown requires a reader backed by a ParquetFileReader, but " - + reader.getClass().getName() + " does not expose one"); - } - if (configuration != null) { - useColumnIndexFilter = configuration.getBoolean( - ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true); + // Late materialization drives a ParquetFileReader directly, so without one the filter is + // simply not applied and the plain read path runs. The conjunct is in the post-scan filter as + // well, so the rows it would have dropped are dropped above the scan. + storageFilter = null; + return; } - + useColumnIndexFilter = configuration.getBoolean( + ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true); // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the // latter can happen under schema evolution: a column is in the requested schema but not in this // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive @@ -747,7 +724,7 @@ private void initializeLateMaterialization() throws IOException { int idx = keyIndices[i]; if (idx < 0 || idx >= parquetColumn.children().size()) { // Unreachable: ParquetStorageFilter.create rejects out-of-range ordinals. - throw new UnsupportedFileReadException(String.format( + throw new IllegalStateException(String.format( "Storage-filter key ordinal %d is out of range for a %d-column requested schema", idx, parquetColumn.children().size())); } @@ -755,7 +732,7 @@ private void initializeLateMaterialization() throws IOException { if (!column.isPrimitive()) { // Unreachable: ParquetStorageFilter.isSupportedKeyType admits only types with a primitive // Parquet leaf, and it gates both planning and ParquetStorageFilter.create. - throw new UnsupportedFileReadException( + throw new IllegalStateException( "Storage-filter key column is not a primitive Parquet column: " + column.path()); } if (missingColumns.contains(column)) { @@ -785,6 +762,7 @@ private void initializeLateMaterialization() throws IOException { if (presentKeyColumns.isEmpty()) { // Every key column is missing, so the rewritten predicate is constant for this file. boolean keepAll = storageFilter.evalAllMissing(); + if (!keepAll) recordFileSkipped(); storageFilter = null; hitEndOfData = !keepAll; return; @@ -807,9 +785,10 @@ private void initializeLateMaterialization() throws IOException { requestedColumns = requestedSchema.getColumns(); keyOnlyColumns = keySchemaBuilder.named(requestedSchema.getName()).getColumns(); - // Build the non-key (complement) schema, which phase 2 reads under finalRanges. When the - // projection is all key columns, phase 2 has nothing to read and is skipped entirely, which a - // null `nonKeyColumns` is what says downstream. + // Build the non-key (complement) schema, which phase 2 reads under finalRanges. A null + // `nonKeyColumns` says there is nothing for phase 2 to read, so it is skipped entirely. The + // planner does not push a scan of that shape, since such a scan reads what a plain one would, + // so this is reachable only by driving the reader directly. Types.MessageTypeBuilder nonKeyBuilder = Types.buildMessage(); int nonKeyFieldCount = 0; for (Type field : requestedSchema.getFields()) { @@ -830,8 +809,9 @@ private void initializeLateMaterialization() throws IOException { /** * Populates the splicing bookkeeping. {@code keyColumnIndices} already index the top-level * slots of {@link #sparkSchema}, the same indexing as {@link #columnVectors}, so they map - * directly onto {@link #isKeyTopLevel}, which {@link #nextBatch()} and {@link #close()} then - * use as the splicing-is-active indicator. + * directly onto {@link #isKeyTopLevel}, which says which batch slots the emit path may take from + * the survivor queues. {@link #initBatch} also reads it, as the sign that this file splices at + * all and needs its own batch array. */ @SuppressWarnings("unchecked") private void initializeSplicingState(List presentKeyColumns) { @@ -862,6 +842,18 @@ private void initializeSplicingState(List presentKeyColumns) { } } + /** + * What phase 2 of the current row group will hold for its row ranges. Every column reader it + * drives materializes the row group's range list of its own ({@code ParquetReadState}), and a + * filter whose survivors are scattered makes one range per surviving row. + */ + private long rowRangeStateBytes(long rangeCount) { + int leaves = spliceCurrentRowGroup + ? (nonKeyColumns == null ? 0 : nonKeyColumns.size()) + : requestedColumns.size(); + return rangeCount * ParquetReadState.ESTIMATED_ROW_RANGE_BYTES * leaves; + } + /** Whether a key value lives in the vector's byte child rather than in its fixed-width array. */ private static boolean isVariableLength(DataType dt) { if (dt instanceof DecimalType decimalType) { @@ -945,8 +937,11 @@ private void checkEndOfRowGroup() throws IOException { * column index (metadata-only) using {@link ParquetFileReader#getRowRanges}. * - Phase 1 (key-only schema): read key-column pages restricted to {@code pushedFilterRanges}, * evaluate the storage filter per row, build {@code finalRanges}. - * - Phase 2 (non-key schema): read non-key columns restricted to {@code finalRanges}. - * Skipped entirely when {@link #nonKeyColumns} is null (all-keys projection). + * - Phase 2: read the non-key columns restricted to {@code finalRanges}. A row group that gave + * splicing up reads the whole projection instead, still under {@code finalRanges}, and one + * that gave the filter up reads it under {@code pushedFilterRanges}, which is what a plain + * scan reads. Skipped entirely only for an all-keys projection that is still splicing, since + * emit then builds every batch from the key queues alone. * * Row groups for which {@code finalRanges} is empty are skipped entirely (no phase-2 IO). * Sets {@link #hitEndOfData} when all row groups have been processed. @@ -962,10 +957,11 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { } // Splicing buffers one key value per surviving row of the whole row group before it can emit // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it - // holds and gives up past the cap. This row group is then read the plain way: phase 2 takes - // every projected column under finalRanges, key columns included, so nothing is buffered and - // the cost is one extra read of the key columns. - spliceCurrentRowGroup = true; + // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold. + // Past that it gives splicing up, and past it again the filter itself, which is what + // `filterGivenUp` says. A file already known to have no offset index starts there. + filterGivenUp = fileHasNoOffsetIndex; + spliceCurrentRowGroup = !filterGivenUp; splicedBytes = 0L; // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full @@ -974,11 +970,10 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { lateMatReader.setRequestedSchema(requestedColumns); // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(), // so calling it unconditionally would keep applying column-index filtering after a user - // turned it off -- the documented escape hatch for files whose column index is wrong. - // Trusting a wrong column index here drops rows for good, since finalRanges is a subset of - // these ranges and the post-scan Filter no longer holds the predicate. Phase 2 is - // unaffected: it selects pages through the offset index, which this conf says nothing - // about. + // turned it off, which is the escape hatch for a file whose column index is wrong. Every + // phase below reads within these ranges, so a wrong column index would cost rows the plain + // path would have returned. Phase 2 is unaffected: it selects pages through the offset index, + // a separate structure this conf says nothing about. RowRanges pushedFilterRanges = useColumnIndexFilter ? lateMatReader.getRowRanges(blockIdx) : RowRanges.createSingle(blockRowCount); @@ -995,11 +990,12 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // a plain read of this projection would transfer for every row the pushed filter kept. The // null checks only skip work for a caller that drives this reader without a scan's metrics; // FileSourceScanLike creates all five whenever storageFilters is non-empty. - // compressedBytesForRowRanges never does IO of its own. + // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already + // given up reports nothing either way, so it does not pay for the baseline at all. StorageFilterMetrics m = storageFilter.metrics(); SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup(); SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering(); - boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null; + boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp; Map blockChunks = needBytes ? chunksByPath(lateMatReader, blockIdx) : null; long nonKeyBaselineBytes = needBytes @@ -1008,40 +1004,49 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { : 0L; // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the - // storage filter per row. - lateMatReader.setRequestedSchema(keyOnlyColumns); - PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges); - if (keyPages == null) { - // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already - // know pushedFilterRanges selects at least one row. Skipping the block here would drop its - // surviving rows from the output, so assert rather than `continue`. - throw new UnsupportedFileReadException( - "No key pages for row group " + blockIdx + " despite " + baselineRows - + " rows selected by the pushed filter"); - } - RowRanges finalRanges = evaluateStorageFilter(keyPages, pushedFilterRanges); - long finalRowCount = finalRanges.rowCount(); - - if (finalRowCount == 0) { - // Every surviving row was rejected by the storage filter; skip block entirely, which avoids - // the whole non-key baseline. Phase 1 still paid to read the key columns, and that cost is - // not part of the baseline, so nothing has to be subtracted from it here. - SQLMetric rgSkipped = m.rowGroupsSkipped(); - if (rgSkipped != null) rgSkipped.add(1L); - SQLMetric rowsExcludedRg = m.rowsExcludedByRowGroup(); - if (rowsExcludedRg != null) rowsExcludedRg.add(baselineRows); - if (bytesAvoidedRg != null) bytesAvoidedRg.add(nonKeyBaselineBytes); - continue; + // storage filter per row. Skipped for a row group the filter is already given up for, which + // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would. + RowRanges finalRanges = pushedFilterRanges; + long finalRowCount = baselineRows; + if (!filterGivenUp) { + lateMatReader.setRequestedSchema(keyOnlyColumns); + PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges); + if (keyPages == null) { + // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already + // know pushedFilterRanges selects at least one row. Skipping the block here would drop + // its surviving rows from the output, so assert rather than `continue`. + throw new IllegalStateException( + "No key pages for row group " + blockIdx + " despite " + baselineRows + + " rows selected by the pushed filter"); + } + RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges); + if (!filterGivenUp + && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) { + // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in + // a single one is only caught here, with its survivors buffered. Those are released, + // since the ranges they were spliced against are about to be thrown away. + giveUpFilter(); + } + if (!filterGivenUp) { + finalRanges = survivors; + finalRowCount = survivors.rowCount(); + if (finalRowCount == 0) { + // Every surviving row was rejected by the storage filter; skip the block entirely, + // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns, + // and that cost is not part of the baseline, so nothing is subtracted from it here. + recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes); + continue; + } + } } - // Phase 2: switch to non-key schema, read non-key columns under finalRanges. Skipped entirely - // when the projection is all keys (nonKeyColumns is null); emit reconstructs each batch from - // the key queues alone. + // Phase 2 reads the non-key columns under the surviving rows, or the whole projection under + // `pushedFilterRanges` for a row group whose filter was given up. It is skipped only when the + // projection is all keys and their values were buffered, since emit then builds every batch + // from the key queues alone. long keptRows; long phase2Bytes; PageReadStore dataPages = null; - // An all-keys projection has nothing for phase 2 to read -- but only if the key values were - // buffered. A row group past the cap has to read them here like any other column. if (nonKeyColumns == null && spliceCurrentRowGroup) { keptRows = finalRowCount; phase2Bytes = 0L; @@ -1052,42 +1057,45 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // enforces that itself: it resolves every requested column's offset index before reading // anything, and a column without one makes its column index store throw // MissingOffsetIndexException. Files written before parquet-mr 1.11, or by a writer that - // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), - // have none. Degrading to a whole-block read is not an option, because the key vectors - // hold only the survivors and the batch would misalign, and neither is dropping the - // predicate, which extraction has removed from the post-scan Filter. So the read fails, - // and all this adds is what the user can do about it. + // omits the page index (pyarrow's `write_table` defaults to `write_page_index=False`), have + // none. The filter is then given up for this row group and the read retried over + // `pushedFilterRanges`, which is what a plain scan reads. That retry cannot hit the same + // wall: a store missing one column's offset index reports no column index either, so + // `getRowRanges` could not have narrowed anything and the ranges cover the whole block. // - // Nothing is checked up front: a row group the filter keeps whole never needs the index - // (`readFilteredRowGroup` falls back to a plain read when the ranges cover the block), and - // one it rejects whole is never read at all, so a file with no page index still scans as - // long as the filter never has to prune inside a row group. + // Nothing is checked up front, so a file with no page index still reads with the filter + // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a + // plain read when the ranges cover the block) or rejects one whole. try { dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); } catch (MissingOffsetIndexException e) { - throw new UnsupportedFileReadException(String.format( - "Storage-filter pushdown needs a Parquet offset index to read the %d of %d rows its " - + "filter kept in row group %d of %s, but the file was written without a page " - + "index. Set %s=false to read this file.", - finalRowCount, blockRowCount, blockIdx, lateMatReader.getFile(), - SQLConf$.MODULE$.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED().key()), e); + LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a " + + "Parquet offset index, and this file was written without a page index for at least " + + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile())); + fileHasNoOffsetIndex = true; + giveUpFilter(); + finalRanges = pushedFilterRanges; + finalRowCount = baselineRows; + lateMatReader.setRequestedSchema(requestedColumns); + dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); } if (dataPages == null) { // Unreachable: readFilteredRowGroup returns null only for an empty block or empty ranges, // both excluded above. Match phase 1 and fail with a message rather than an NPE. - throw new UnsupportedFileReadException( + throw new IllegalStateException( "No data pages for row group " + blockIdx + " despite " + finalRowCount - + " surviving rows"); + + " rows to read"); } keptRows = dataPages.getRowCount(); - // `needBytes`, not just `bytesAvoidedPf != null`: it is what built `blockChunks`. - if (needBytes && bytesAvoidedPf != null) { + // Nothing is computed for a row group whose filter was given up: it read what a plain scan + // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`, + // because that is what built `blockChunks`. + if (needBytes && bytesAvoidedPf != null && !filterGivenUp) { phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns, finalRanges, finalRowCount); if (!spliceCurrentRowGroup) { // This row group gave splicing up, so phase 2 read the key columns a second time. The - // baseline counts them once, in phase 1, so the extra read is a cost against it -- - // which can make the row group's contribution negative, and that is the truth about it. + // baseline counts them once, in phase 1, so the extra read is a cost against it. phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, keyOnlyColumns, finalRanges, finalRowCount); } @@ -1098,7 +1106,9 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { long filteredRows = baselineRows - keptRows; SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup(); if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows); - if (bytesAvoidedPf != null) { + if (bytesAvoidedPf != null && !filterGivenUp) { + // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline + // after giving splicing up contributes nothing rather than subtracting. bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes); } @@ -1121,6 +1131,53 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { } + /** Counts a row group whose data columns the filter kept the reader from touching at all. */ + private static void recordRowGroupSkipped( + StorageFilterMetrics m, long excludedRows, long avoidedBytes) { + SQLMetric rgSkipped = m.rowGroupsSkipped(); + if (rgSkipped != null) rgSkipped.add(1L); + SQLMetric rowsExcluded = m.rowsExcludedByRowGroup(); + if (rowsExcluded != null) rowsExcluded.add(excludedRows); + SQLMetric bytesAvoided = m.bytesAvoidedByRowGroup(); + if (bytesAvoided != null) bytesAvoided.add(avoidedBytes); + } + + /** + * Counts a file the filter rejects whole, which happens when every key column is missing from it + * and the predicate is constant-false for the value the reader would have materialized. Every row + * group counts as skipped and every projected byte as avoided, which is what the counters mean + * for a row group the filter empties. + */ + private void recordFileSkipped() { + StorageFilterMetrics m = storageFilter.metrics(); + boolean needBytes = m.bytesAvoidedByRowGroup() != null; + if (m.rowGroupsSkipped() == null && m.rowsExcludedByRowGroup() == null && !needBytes) return; + List projected = requestedSchema.getColumns(); + List blocks = lateMatReader.getRowGroups(); + for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) { + // Measured against the rows the pushed data filter kept, which is the baseline every other + // skip path uses: the rows its column index already excluded were never this filter's to + // save. `getRowRanges` is a cache hit whenever the two can differ, because + // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be + // guarded the same way phase 0 guards it, since it consults the pushed filter but not the + // conf that turns column-index filtering off. + long blockRowCount = blocks.get(blockIdx).getRowCount(); + if (blockRowCount == 0) continue; + RowRanges blockRanges = useColumnIndexFilter + ? lateMatReader.getRowRanges(blockIdx) + : RowRanges.createSingle(blockRowCount); + long survivingRows = blockRanges.rowCount(); + if (survivingRows == 0) continue; + // The key columns are missing from this file, so they contribute nothing to the walk, and the + // whole projection is what a plain read would have transferred. + long avoidedBytes = needBytes + ? compressedBytesForRowRanges(lateMatReader, blockIdx, + chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows) + : 0L; + recordRowGroupSkipped(m, survivingRows, avoidedBytes); + } + } + /** * The block's column chunks by path, built once per row group and shared by the byte-metric calls * that consume it, since {@link BlockMetaData} offers no lookup of its own. @@ -1143,13 +1200,15 @@ private static Map chunksByPath( *

Two sources, chosen so this never causes IO of its own: *

    *
  • {@code rowRanges} covers the whole block: the answer is the sum of the chunks' - * {@code getTotalSize()}, which is already in the footer. This is the case that matters -- + * {@code getTotalSize()}, which is already in the footer. This is the case that matters: * whenever nothing else has built the block's {@link ColumnIndexStore}, {@code rowRanges} * is necessarily the whole block, because a narrower range can only come from column-index * filtering, which builds the store as a side effect. *
  • {@code rowRanges} is a strict subset: walk the offset index, as parquet's own read path * does, and add the dictionary page the way {@code calculateOffsetRanges} does. The store - * is guaranteed to exist here, so the walk is pure metadata arithmetic. + * is guaranteed to exist here, so the walk is pure metadata arithmetic. For the ranges the + * storage filter narrowed, which column-index filtering had no hand in, that guarantee is + * an ordering one: phase 2's own read of those ranges built the store first. *
* *

Columns absent from this physical file (schema evolution) contribute nothing, which is @@ -1225,6 +1284,9 @@ private static long dictionaryPageSize(ColumnChunkMetaData chunk) { * to splice, until the buffer passes its cap. From there the row group is evaluated without * buffering and {@link #spliceCurrentRowGroup} is false, so its phase 2 reads the key columns * again along with everything else. + * + *

Returns null once the budget makes the reader give the filter up for this row group: the + * ranges built so far are then incomplete, and the caller reads the row group the plain way. */ private RowRanges evaluateStorageFilter( PageReadStore keyPages, @@ -1240,6 +1302,8 @@ private RowRanges evaluateStorageFilter( PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator(); RowRanges.Builder finalRangesBuilder = RowRanges.builder(); + survivorRangeCount = 0L; + long previousSurvivor = -2L; // Recomputed rather than taken from the caller: a count that disagreed with this iterator would // silently drop surviving rows, and no post-scan Filter is left to catch that. long remaining = pushedFilterRanges.rowCount(); @@ -1255,8 +1319,12 @@ private RowRanges evaluateStorageFilter( long blockRow = rowIndexIter.nextLong(); if (storageFilter.test(keyScratchBatch.getRow(r))) { finalRangesBuilder.addSelectedRow(blockRow); + if (blockRow != previousSurvivor + 1) survivorRangeCount++; + previousSurvivor = blockRow; if (accumulate) { accumulate = appendSurvivorRowToAccumulators(r); + // The ranges being built are about to be thrown away, so stop evaluating the rest. + if (filterGivenUp) return null; } } } @@ -1338,11 +1406,20 @@ private boolean appendSurvivorRowToAccumulators(int srcRow) { keyVectorQueues[i].addLast(currentKeyAccumulators[i]); currentKeyAccumulators[i] = null; } - // Only here, so the cost is one comparison per capacity-sized vector rather than per row. A - // row group whose survivors fit in a single accumulator is never checked at all: it then - // holds one capacity-sized vector per key column, which is what a plain read holds anyway. - if (splicedBytes > storageFilter.maxSplicedRowGroupBytes()) { + // Checked only here, so the cost is one comparison per capacity-sized vector rather than per + // row. A row group whose survivors fit in a single accumulator is never checked at all: it + // then holds one capacity-sized vector per key column, which is what a plain read holds. + // + // Two allocations share the budget, and giving splicing up releases only the first, so the + // cheaper concession comes first: drop the buffer, and give the filter up as well if the row + // ranges alone still do not fit. The second measurement is taken after that concession, so it + // counts the leaves phase 2 will now drive, which is every projected column rather than the + // non-key ones. Either way this row group stops accumulating. + long cap = storageFilter.maxSplicedRowGroupBytes(); + if (splicedBytes + rowRangeStateBytes(survivorRangeCount) > cap) { abandonSplicing(); + // Splicing is already gone, so the flag is all that is left to set. + if (rowRangeStateBytes(survivorRangeCount) > cap) filterGivenUp = true; return false; } ensureCurrentKeyAccumulatorsAllocated(); @@ -1350,6 +1427,16 @@ private boolean appendSurvivorRowToAccumulators(int srcRow) { return true; } + /** + * Gives the filter up for the row group being read, which leaves phase 2 reading every projected + * column over the rows the pushed data filter allowed, exactly what a plain read does. Splicing + * goes with it: the buffered survivors are no longer the rows that will be emitted. + */ + private void giveUpFilter() { + filterGivenUp = true; + abandonSplicing(); + } + /** * Gives up splicing for the row group being evaluated and releases every survivor vector it has * buffered. Phase 2 then reads the full projected schema and the emit path takes the persistent @@ -1417,7 +1504,7 @@ private interface ValueCopier { /** * Returns a {@link ValueCopier} for the given key {@link DataType}. The set of types handled here * is the definition behind {@code ParquetStorageFilter.isSupportedKeyType}, which gates both - * planning-time extraction and {@code ParquetStorageFilter.create} -- so the throw at the end is + * planning-time extraction and {@code ParquetStorageFilter.create}, so the throw at the end is * unreachable. Teach both sides at once when adding a type; a type admitted there but missing * here becomes a task failure instead of a planning-time rejection. */ @@ -1463,7 +1550,7 @@ private static ValueCopier copierFor(DataType dt) { if (dt instanceof StringType || dt instanceof BinaryType) { return (dst, dRow, src, sRow) -> dst.putByteArray(dRow, src.getBinary(sRow)); } - throw new UnsupportedFileReadException( + throw new IllegalStateException( "Splicing storage-filter pushdown does not support key type: " + dt); } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index 6b281c9e8dbc6..aa5fead84b538 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -304,8 +304,9 @@ trait FileSourceScanLike extends DataSourceScanExec with SessionStateHelper { // Filters on non-partition columns. def dataFilters: Seq[Expression] // Filters the storage layer evaluates to prune value-column IO based on key-column evaluation. - // These may reference subqueries (e.g. a runtime bloom filter built from a join build side) and - // are materialized at task launch time. Nil for a scan that does not support pushing them. + // These may reference subqueries (e.g. a runtime bloom filter built from a join build side), + // which are materialized on the driver while the RDD is built, before the reader is serialized. + // Nil for a scan that does not support pushing them. def storageFilters: Seq[Expression] = Nil // Disable bucketed scan based on physical query plan, see rule // [[DisableUnnecessaryBucketedScan]] for details. @@ -757,7 +758,7 @@ object FileSourceScanLike { * [[DisableUnnecessaryBucketedScan]] for details. * @param storageFilters Filters evaluated by the storage layer (e.g. parquet reader) to drive * value-column IO pruning based on key-column evaluation. May contain - * subqueries materialized at task launch. + * subqueries, which `preparedStorageFilters` materializes on the driver. */ case class FileSourceScanExec( @transient override val relation: HadoopFsRelation, @@ -840,9 +841,8 @@ case class FileSourceScanExec( if (storageFilters.isEmpty) { Nil } else { - // No conf check here: extraction has already removed these conjuncts from the post-scan - // Filter, so re-reading the conf would drop the filter for good if the user turned it off - // between planning and execution. The conf gates extraction at planning time only. + // No conf check here: the conf decides at planning time whether a scan is offered storage + // filters at all, and re-reading it now could only make this scan drop work it already has. // // `output` is `readDataColumns ++ generatedMetadataColumns ++ partitionColumns ++ // constantMetadataColumns` and `requiredSchema` is the StructType of the first two groups, so diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala index 8662ce7ebd03b..2eb4d04c7d897 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala @@ -269,15 +269,7 @@ object DataSourceUtils extends PredicateHelper { QueryExecutionErrors.sparkUpgradeInWritingDatesError(format, config) } - /** - * Whether `ignoreCorruptFiles` may swallow this failure and skip the rest of its file. - * - * [[UnsupportedFileReadException]] is excluded because it does not report a corrupt file: it says - * the file cannot support a read the plan depends on. Skipping the rest of such a file would drop - * rows that are perfectly readable, and would do it silently. - */ def shouldIgnoreCorruptFileException(e: Throwable): Boolean = e match { - case _: UnsupportedFileReadException => false case _: RuntimeException | _: IOException | _: InternalError => true case _ => false } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala index 9a20bdcd37928..3f58fb6c7cad6 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala @@ -171,15 +171,17 @@ trait FileFormat { * Catalyst expressions that the storage layer may evaluate to drive value-column IO pruning based * on key-column evaluation (e.g., late materialization with a runtime bloom filter). * - * The default implementation delegates to [[buildReaderWithPartitionValues]] and accepts no - * storage filter at all. A format that supports storage-filter pushdown overrides this, and must - * not let the two builders call each other: this default delegates one way, so an override that - * delegates back recurses until the driver's stack runs out, `super` included, since that call is - * virtual too. Route both to a private implementation instead, the way `ParquetFileFormat` does. + * Honoring them is optional, here and in a reader that does implement them: the planner leaves + * every one of them in the post-scan `Filter` as well, so ignoring one is a missed optimization + * rather than a wrong answer. That is why this default can simply delegate to + * [[buildReaderWithPartitionValues]]. * - * A non-empty `storageFilters` here is a planner bug, so the default body rejects it rather than - * dropping it: the planner removes an extracted conjunct from the post-scan `Filter`, so a reader - * that ignores it returns rows the filter rejects. + * A format that supports storage-filter pushdown overrides this, and must not let the two + * builders call each other. This default delegates one way, and it delegates on `this`, so an + * override of [[buildReaderWithPartitionValues]] that delegates back here closes the loop and + * recurses until the driver's stack runs out. Calling this default through `super` is part of + * that loop, not an escape from it. Route both to a private implementation instead, the way + * `ParquetFileFormat` does. * * Scalar subqueries inside `storageFilters` are expected to have been materialized before this * method is called, so that the returned reader can be safely serialized to executors. @@ -199,21 +201,19 @@ trait FileFormat { hadoopConf: Configuration, storageFilterMetrics: Map[String, SQLMetric] = Map.empty ): PartitionedFile => Iterator[InternalRow] = { - require(storageFilters.isEmpty, - s"${getClass.getSimpleName} does not support storage-filter pushdown, but was given " + - storageFilters.mkString("[", ", ", "]")) buildReaderWithPartitionValues( sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf) } /** * Whether this format's reader can evaluate `expr` as a storage filter, i.e. whether the planner - * may extract it from the post-scan `Filter` and hand it to [[buildReaderWithStorageFilters]]. + * may offer it to [[buildReaderWithStorageFilters]]. * - * The planner decides what it can see from the plan -- that the conjunct is deterministic and - * references only projected data columns -- and asks this for everything else, so the expression - * shapes and column types a reader supports stay in that reader's own package. A format that - * answers true for an expression must be able to honor it: the planner removes it from the plan. + * The planner decides what it can see from the plan, that the conjunct is deterministic and + * references only projected data columns. It asks this for everything else, so the expression + * shapes and column types a reader supports stay in that reader's own package. Answering true + * says the reader can evaluate the expression, not that it will: the conjunct stays in the + * post-scan `Filter`, so a reader is free to give a file up. */ def supportsStorageFilter(expr: Expression): Boolean = false diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala index db92448468f9f..300f0d80a10d8 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala @@ -152,19 +152,23 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { } /** - * Splits `afterScanFilters` into conjuncts the file format can evaluate at the storage layer for - * late materialization (returned as the first element) and the remaining filters that stay as a - * post-scan FilterExec (returned as the second element). + * The conjuncts of `afterScanFilters` the file format can evaluate at the storage layer for late + * materialization, to prune value-column IO. * - * Everything is checked here, statically, so the runtime never silently loses a filter. Two of - * the conditions are per scan, and failing either leaves every filter in the plan: + * They stay in the post-scan `Filter` as well, the way a pushed data filter does: the reader is + * offered them, not obliged to honor them, so the plan keeps the exact check. What it costs is + * evaluating the conjunct a second time for the rows the scan emits. What it buys is a reader + * free to give up on a file with no page index, or on a row group whose survivors scatter too far + * to hold their row ranges, without the answer depending on it. + * + * Two of the conditions are per scan, and failing either offers nothing: * - The storage-filter pushdown SQL conf is on. * - [[FileFormat.supportBatch]] holds for the schema the reader will see, * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives * its own vectorized-read decision from. Late materialization needs a batch read, so this asks * about batch support rather than naming a format. * - * The rest are per conjunct, and one that fails any of them stays in the post-scan Filter: + * The rest are per conjunct: * - It is deterministic. A reader evaluates the predicate without * `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a * `Nondeterministic` expression, so a non-deterministic conjunct would fail at task time. @@ -176,33 +180,28 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { * One last condition is on the set that survives: at least one projected data column must be left * for the reader to prune. A scan that projects nothing but the filter's own key columns reads * the same columns for the same rows either way, since the reader has to read a key column to - * evaluate the filter on it, so pushing can only add the cost of evaluating the predicate outside - * the generated code. Extraction is dropped entirely in that case. + * evaluate the filter on it, so offering it could only add the cost of evaluating the predicate + * outside the generated code. Nothing is offered in that case. */ - private def extractStorageFilters( + private def storageFiltersFor( afterScanFilters: ExpressionSet, fsRelation: HadoopFsRelation, readDataColumns: Seq[Attribute], - outputDataSchema: StructType): (Seq[Expression], ExpressionSet) = { + outputDataSchema: StructType): Seq[Expression] = { val sparkSession = fsRelation.sparkSession val sqlConf = sparkSession.sessionState.conf - if (!sqlConf.parquetStorageFilterPushdownEnabled) return (Nil, afterScanFilters) + if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields) - if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) { - return (Nil, afterScanFilters) - } + if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil val dataAttrs = AttributeSet(readDataColumns) - val (eligible, rest) = afterScanFilters.partition { expr => + val offered = afterScanFilters.filter { expr => val refs = expr.references expr.deterministic && refs.nonEmpty && refs.forall(dataAttrs.contains) && fsRelation.fileFormat.supportsStorageFilter(expr) - } - val keyAttrs = AttributeSet(eligible.toSeq.flatMap(_.references)) - if (eligible.isEmpty || readDataColumns.forall(keyAttrs.contains)) { - return (Nil, afterScanFilters) - } - (eligible.toSeq, ExpressionSet(rest)) + }.toSeq + val keyAttrs = AttributeSet(offered.flatMap(_.references)) + if (readDataColumns.forall(keyAttrs.contains)) Nil else offered } def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match { @@ -272,10 +271,6 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { logInfo(log"Post-Scan Filters: ${MDC(POST_SCAN_FILTERS, afterScanFilters.simpleString(maxToStringFields))}") - // `filterAttributes` is computed from `afterScanFilters` before storage-filter extraction, - // which happens further down once `outputDataSchema` is known, so a column referenced only by - // an extracted filter is still in `requiredAttributes` and survives projection pruning. The - // reader has to read it to evaluate the filter. val filterAttributes = AttributeSet(afterScanFilters ++ stayUpFilters) val requiredExpressions: Seq[NamedExpression] = filterAttributes.toSeq ++ projects val requiredAttributes = AttributeSet(requiredExpressions) @@ -353,11 +348,11 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { val outputDataSchema = (readDataColumns ++ generatedMetadataColumns).toStructType - // Extracted conjuncts become `storageFilters` on the scan and leave the post-scan Filter, - // since the reader applies them exactly. This runs here rather than next to - // `afterScanFilters` because eligibility depends on `outputDataSchema`. - val (storageFilters, remainingAfterScanFilters) = extractStorageFilters( - afterScanFilters, fsRelation, readDataColumns, outputDataSchema) + // Offered conjuncts become `storageFilters` on the scan and stay in the post-scan Filter too. + // This runs here rather than next to `afterScanFilters` because eligibility depends on + // `outputDataSchema`. + val storageFilters = + storageFiltersFor(afterScanFilters, fsRelation, readDataColumns, outputDataSchema) // The output rows will be produced during file scan operation in three steps: // (1) File format reader populates a `Row` with `readDataColumns` and @@ -424,8 +419,7 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { }.getOrElse(scan) // bottom-most filters are put in the left of the list. - val finalFilters = - remainingAfterScanFilters.toSeq.reduceOption(expressions.And).toSeq ++ stayUpFilters + val finalFilters = afterScanFilters.toSeq.reduceOption(expressions.And).toSeq ++ stayUpFilters val withFilter = finalFilters.foldLeft(withMetadataProjections)((plan, cond) => { execution.FilterExec(cond, plan) }) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala deleted file mode 100644 index cabc5e416c2f5..0000000000000 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/UnsupportedFileReadException.scala +++ /dev/null @@ -1,35 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.spark.sql.execution.datasources - -/** - * Thrown when a file cannot support a read the plan depends on, as opposed to being corrupt or - * missing. The distinction matters because `ignoreCorruptFiles` asks Spark to skip the rest of a - * file whose read failed, and [[DataSourceUtils.shouldIgnoreCorruptFileException]] excludes this - * exception for that reason: skipping here would silently drop rows of a perfectly good file. - * - * A reader throws this when the plan above it assumes the reader does something the file cannot - * express. Storage-filter pushdown is the case that motivated it: the planner removes the pushed - * conjunct from the post-scan `Filter`, so a reader that cannot apply it has to fail the query - * rather than return rows the filter rejects. - */ -class UnsupportedFileReadException(message: String, cause: Throwable) - extends RuntimeException(message, cause) { - - def this(message: String) = this(message, null) -} diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index 0118fb8e1aeb9..eaa47ac395302 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -222,7 +222,7 @@ class ParquetFileFormat /** * The implementation behind both public entry points above, which is why neither of them calls - * the other -- see the warning on `FileFormat.buildReaderWithStorageFilters`. + * the other. See the warning on `FileFormat.buildReaderWithStorageFilters`. */ private def buildParquetReader( sparkSession: SparkSession, @@ -274,22 +274,16 @@ class ParquetFileFormat val int96RebaseModeInRead = parquetOptions.int96RebaseModeInRead val archiveFormatEnabled = parquetOptions.archiveFormatEnabled - // Extraction has already removed these conjuncts from the post-scan Filter, so nothing else in - // the plan would apply them: anything that stops the reader from honoring them fails loudly - // rather than dropping them. `enableVectorizedReader` is one such thing, and it is recomputed - // from the live session conf when the RDD is built, so a flip of + // Late materialization needs the vectorized reader. `enableVectorizedReader` is recomputed from + // the live session conf when the RDD is built, so a flip of // spark.sql.parquet.enableVectorizedReader (or the nested-column variant) after planning lands - // here. + // here, and the filters are simply not installed: the post-scan Filter still holds them. val storageFilterOpt: Option[ParquetStorageFilter] = if (storageFilters.isEmpty) { None } else if (!enableVectorizedReader) { - throw new UnsupportedFileReadException( - "Cannot honor storage filters " + storageFilters.mkString("[", ", ", "]") + - " because the " + - "vectorized Parquet reader is disabled for schema " + resultSchema.catalogString + ". " + - "The scan was planned with storage-filter pushdown, which requires the vectorized " + - "reader; a vectorized-reader conf was most likely changed after the query was planned. " + - s"Set ${SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key}=false and re-run the query.") + logInfo(log"Not honoring storage filters for schema " + + log"${MDC(SCHEMA, resultSchema.catalogString)}: the vectorized Parquet reader is disabled") + None } else { val metrics = StorageFilterMetrics( rowGroupsSkipped = storageFilterMetrics.getOrElse( @@ -302,8 +296,8 @@ class ParquetFileFormat FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, null), bytesAvoidedByPageFiltering = storageFilterMetrics.getOrElse( FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null)) - // `create` requires every condition extractStorageFilters already pre-checked, so it throws - // rather than letting us drop the filter. + // `create` requires every condition extractStorageFilters already pre-checked, so a violation + // is a planner bug rather than something to work around here. Some(ParquetStorageFilter.create(storageFilters, requiredSchema, metrics, sqlConf.parquetStorageFilterPushdownMaxSplicedRowGroupBytes)) } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala index b578ba8d0042c..e7d27fa863842 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala @@ -17,8 +17,10 @@ package org.apache.spark.sql.execution.datasources.parquet +import java.util.concurrent.ConcurrentHashMap + import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate} +import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, ExprUtils, Literal, Predicate, XxHash64} import org.apache.spark.sql.execution.metric.SQLMetric import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType} @@ -70,6 +72,14 @@ class ParquetStorageFilter private ( // construction to first use on the executor. @transient private lazy val predicate: BasePredicate = Predicate.create(boundExpression) + // Rewrites are cached for the task, because `rewriteForMissingKeys` rebuilds the expression with + // `transform`, and a rebuilt `BloomFilterMightContain` has to deserialize its filter again, up to + // megabytes per file otherwise. The missing positions are the whole key: the values a caller + // substitutes for them are the ones the scan's schema defines, so every file of that scan passes + // the same ones. + @transient private lazy val rewrites = + new ConcurrentHashMap[Seq[Int], ParquetStorageFilter]() + def test(keyRow: InternalRow): Boolean = predicate.eval(keyRow) /** @@ -80,7 +90,9 @@ class ParquetStorageFilter private ( * their original relative order, and SQL metrics are shared with `this`. * * `missingKeyValues(i)` must be the internal-format value the reader produces for a missing - * column: its existence DEFAULT when it has one, else null. `ParquetColumnVector` writes that + * column: its existence DEFAULT when it has one, else null. The result is cached per set of + * missing positions, so those values have to be a function of the positions, which they are: they + * come from the scan's schema. `ParquetColumnVector` writes that * default into the output vector, so substituting null instead would filter on a value the scan * never returns and could drop matching rows. * @@ -96,6 +108,13 @@ class ParquetStorageFilter private ( missingKeyValues: Array[Any]): ParquetStorageFilter = { require(missingKeyLocalPositions.length == missingKeyValues.length, "missingKeyLocalPositions and missingKeyValues must have the same length") + rewrites.computeIfAbsent(missingKeyLocalPositions.toSeq, + _ => rewrite(missingKeyLocalPositions, missingKeyValues)) + } + + private def rewrite( + missingKeyLocalPositions: Array[Int], + missingKeyValues: Array[Any]): ParquetStorageFilter = { val substitution = missingKeyLocalPositions.zip(missingKeyValues).toMap val presentPositions = keyColumnIndices.indices.filterNot(substitution.contains) val newPosOf = presentPositions.zipWithIndex.toMap @@ -128,11 +147,9 @@ object ParquetStorageFilter { * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must * satisfy all of them to survive. * - * Every condition below is a hard precondition rather than a soft rejection. By the time this is - * called, `FileSourceStrategy.extractStorageFilters` has removed these conjuncts from the - * post-scan `Filter`, so nothing left in the plan would apply them; returning a "no filter" - * result would silently return rows the filter rejects. `extractStorageFilters` pre-checks all of - * it, so a violation here is a planner bug and failing is the only safe response. + * Every condition below is asserted rather than handled: `extractStorageFilters` pre-checks all + * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a + * different matter. These conditions are about the filter being well formed at all. * * Callers that have no storage filters must not call this at all. */ @@ -153,11 +170,14 @@ object ParquetStorageFilter { // not go through the remapping: it pairs the k-th key slot of the output batch with key-row // position k, which is the identity only while this list is ascending. val originalOrdinals = expr.collect { case b: BoundReference => b.ordinal }.distinct.sorted + // These messages name the expression by its node rather than printing it: a prepared bloom + // holds its filter as a binary literal, which renders as megabytes of hex. require(originalOrdinals.nonEmpty, - s"storage filter $expr has no bound reference to a key column") + s"storage filter ${expr.prettyName} has no bound reference to a key column") require(originalOrdinals.forall(i => i >= 0 && i < requestedSchema.length), - s"storage filter $expr references ordinals ${originalOrdinals.mkString("[", ", ", "]")} " + - s"outside the ${requestedSchema.length} fields of ${requestedSchema.catalogString}") + s"storage filter ${expr.prettyName} references ordinals " + + s"${originalOrdinals.mkString("[", ", ", "]")} outside the ${requestedSchema.length} " + + s"fields of ${requestedSchema.catalogString}") val unsupported = originalOrdinals.map(requestedSchema.fields(_)) .filterNot(field => isSupportedKeyType(field.dataType)) require(unsupported.isEmpty, @@ -183,11 +203,11 @@ object ParquetStorageFilter { * * Narrower than `AtomicType`, for two different reasons: * - `VariantType` cannot be supported: its Parquet representation is a group, not a primitive - * leaf, so phase 1 has nothing flat to read it into. (It is unreachable anyway -- + * leaf, so phase 1 has nothing flat to read it into. (It is unreachable anyway, since * `HashExpression.checkInputDataTypes` rejects variant, so no bloom can be built on one.) - * - `GeometryType` and `GeographyType` could be supported -- both map to a primitive Parquet + * - `GeometryType` and `GeographyType` could be supported. Both map to a primitive Parquet * BINARY and both are handled by `WritableColumnVector.isArray`, so the existing byte-array - * copier would work -- but no bloom can currently reference them: `HashExpression`'s codegen + * copier would work. But no bloom can currently reference them: `HashExpression`'s codegen * type dispatch has no case for either, so hashing one fails at codegen. They are left out * until something can actually produce such a filter. */ @@ -211,7 +231,29 @@ object ParquetStorageFilter { // NOT: the reader evaluates the expression it is given and treats a false as "drop this row". // Every reference is checked, not just the ones on the value side, because [[create]] binds // and type-checks all of them. - bloom.references.forall(a => isSupportedKeyType(a.dataType)) + bloom.references.forall(a => isSupportedKeyType(a.dataType)) && canEvaluateOnEveryRow(bloom) case _ => false } + + /** + * Whether the reader may evaluate `bloom`'s value side on any row of the files it reads. + * + * It has to ask, because the reader evaluates the predicate on every row the pushed data filter + * left, while in the plan the conjunct ran after the ones ahead of it and was skipped for the + * rows they rejected. A `CAST(s AS BIGINT)` key, which `InjectRuntimeFilter` builds for a + * string-to-long join, then throws in ANSI mode on a row an earlier conjunct would have dropped, + * a query that succeeds without this feature. + * + * So the value side must be a hash of expressions that cannot fail on any input, which + * [[ExprUtils.canEvaluateUnconditionally]] decides. `XxHash64` and the membership test itself are + * total for every type they accept at analysis time. + * + * This is a question about the conjunct as the planner holds it, with attribute references for + * leaves. It is not one to ask of a bound expression, which that whitelist does not admit. + */ + private def canEvaluateOnEveryRow(bloom: BloomFilterMightContain): Boolean = + bloom.valueExpression match { + case hash: XxHash64 => hash.children.forall(ExprUtils.canEvaluateUnconditionally) + case _ => false + } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala index 5243b728988a9..65aa8d8b9240a 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala @@ -28,15 +28,16 @@ import scala.jdk.CollectionConverters._ import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem} import org.apache.hadoop.mapreduce.Job -import org.apache.parquet.hadoop.{ParquetInputFormat, ParquetOutputFormat} +import org.apache.parquet.column.Encoding +import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat} import org.apache.spark.paths.SparkPath -import org.apache.spark.sql.{sources, QueryTest, Row, SparkSession} +import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession} import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Coalesce, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, XxHash64} +import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64} import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter} import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec} -import org.apache.spark.sql.execution.datasources.{DataSourceUtils, FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile, UnsupportedFileReadException} +import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile} import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} import org.apache.spark.sql.functions.col import org.apache.spark.sql.internal.SQLConf @@ -54,20 +55,22 @@ import org.apache.spark.util.sketch.BloomFilter class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { import testImplicits._ - // Writes a parquet file with the given rows and row-group size; returns the path. - private def writeParquetFile( + // Writes `df` as one parquet file under a fresh directory and returns its path. Every write + // helper in this suite goes through here. + private def writeSingleParquetFile( dir: File, - rows: Seq[(Long, String)], - rowGroupSize: Long = 1024L, - pageSize: Option[Long] = None): String = { + df: DataFrame, + rowGroupSize: Long, + pageSize: Option[Long] = None, + dictionary: Boolean = false): String = { val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath - val writer = rows.toDF("k", "v") + val writer = df .repartition(1) .write .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) // Dictionary encoding off keeps row-group sizing predictable. The column index is still // written either way. - .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") + .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString) // A small page size gives each row group several pages per column, which is what lets // column-index filtering produce a row range narrower than the whole row group. pageSize.foreach(size => writer.option(ParquetOutputFormat.PAGE_SIZE, size)) @@ -77,38 +80,20 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { files(0).getAbsolutePath } - // Collects all rows from a reader initialized with the given storage filter. - // - // `tryInitializeResource` closes the reader if anything inside throws and leaves it open - // otherwise, which is the contract these helpers need: the caller closes it once its assertions - // pass. Without it a failure in the read loop -- what these tests are looking for -- would leak - // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade - // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at - // construction. + // Writes a parquet file with the given rows and row-group size; returns the path. + private def writeParquetFile( + dir: File, + rows: Seq[(Long, String)], + rowGroupSize: Long = 1024L, + pageSize: Option[Long] = None): String = + writeSingleParquetFile(dir, rows.toDF("k", "v"), rowGroupSize, pageSize) + + // Collects all `(k, v)` rows from a reader initialized with the given storage filter. private def readAll( filePath: String, - storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = { - Utils.tryInitializeResource { - new VectorizedParquetRecordReader(false, 4096) - } { reader => - reader.setStorageFilter(storageFilter) - reader.initialize(filePath, java.util.Arrays.asList("k", "v")) - reader.initBatch(new StructType(), null) - val collected = mutable.ArrayBuffer[(Long, String)]() - while (reader.nextBatch()) { - val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] - val n = batch.numRows() - val kVec = batch.column(0) - val vVec = batch.column(1) - var i = 0 - while (i < n) { - collected += ((kVec.getLong(i), vVec.getUTF8String(i).toString)) - i += 1 - } - } - (collected.toSeq, reader) - } - } + storageFilter: ParquetStorageFilter): (Seq[(Long, String)], VectorizedParquetRecordReader) = + readAllWith(filePath, Seq("k", "v"), storageFilter, + (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString)) // Builds a `k >= threshold` storage filter bound to position 0. private def keyAtLeastFilter( @@ -125,18 +110,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { private def writeKeyOnlyParquetFile[T : org.apache.spark.sql.Encoder]( dir: File, keys: Seq[T], - rowGroupSize: Long = 1024L): String = { - val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath - spark.createDataset(keys).toDF("k") - .repartition(1) - .write - .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) - .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") - .parquet(outDir) - val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")) - assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir") - files(0).getAbsolutePath - } + rowGroupSize: Long = 1024L): String = + writeSingleParquetFile(dir, spark.createDataset(keys).toDF("k"), rowGroupSize) // Reads a key-only file, returning the survivor keys and the reader. The {@code extract} function // pulls one value at a time from the batch's key column. @@ -144,27 +119,9 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { filePath: String, storageFilter: ParquetStorageFilter, extract: (org.apache.spark.sql.vectorized.ColumnVector, Int) => T, - capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) = { - Utils.tryInitializeResource { - new VectorizedParquetRecordReader(false, capacity) - } { reader => - reader.setStorageFilter(storageFilter) - reader.initialize(filePath, java.util.Arrays.asList("k")) - reader.initBatch(new StructType(), null) - val collected = mutable.ArrayBuffer[T]() - while (reader.nextBatch()) { - val batch = reader.resultBatch().asInstanceOf[ColumnarBatch] - val n = batch.numRows() - val kVec = batch.column(0) - var i = 0 - while (i < n) { - collected += extract(kVec, i) - i += 1 - } - } - (collected.toSeq, reader) - } - } + capacity: Int = 4096): (Seq[T], VectorizedParquetRecordReader) = + readAllWith(filePath, Seq("k"), storageFilter, (batch, i) => extract(batch.column(0), i), + capacity) // Builds a `k >= threshold` storage filter bound to position 0 against a key-only schema of the // given key type. @@ -179,7 +136,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("rejects entire row group: no data-column IO, row-group-skipped metric incremented") { withTempDir { dir => - // 40 rows, ~small row group => >= 2 row groups. + // 40 rows in one row group: parquet's first row-group size check is at record 100, so + // `rowGroupSize` cannot split a file this small. The one row group is the one rejected. val rows = (1L to 40L).map(i => (i, s"v_$i")) val path = writeParquetFile(dir, rows, rowGroupSize = 256L) @@ -243,10 +201,13 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val path = writeParquetFile(dir, rows, rowGroupSize = 256L) val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") // k >= 195 keeps only the last 6 rows; earlier row groups should be skipped. val filter = keyAtLeastFilter(195L, StorageFilterMetrics( - rowGroupsSkipped = rgSkipped, rowsExcludedWithinRowGroup = rowsExcludedPf)) + rowGroupsSkipped = rgSkipped, + rowsExcludedByRowGroup = rowsExcludedRg, + rowsExcludedWithinRowGroup = rowsExcludedPf)) val (result, reader) = readAll(path, filter) try { // Output is exact: VectorizedColumnReader uses PageReadStore.getRowIndexes (driven by @@ -256,6 +217,14 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { s"expected exact filtering; got ${result.map(_._1).sorted}, " + s"expected ${expected.map(_._1).toSeq.sorted}") assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}") + // "Partially kept" is the `rowsExcludedWithinRowGroup` half of the accounting, and only + // this identity establishes it: a row group that was neither skipped whole nor emitted has + // to have its rows counted there. + assert(rowsExcludedPf.value > 0, + s"expected rows excluded inside a kept row group; got ${rowsExcludedPf.value}") + assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == rows.size, + s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " + + s"should account for all ${rows.size} rows") } finally { reader.close() } @@ -267,18 +236,21 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // (`nonKeyColumns == null` in the reader). All output rows come from the per-key-column // queues populated in phase 1. Total bytes read match the no-storage-filter path (phase 1 reads // the key column once instead of phase 2 re-reading it), so both `avoided` metrics are zero: - // there are no non-key bytes to skip. + // there are no non-key bytes to skip. This is the shape the design notes call the biggest win, + // so it must not be the shape that pays for metrics. withTempDir { dir => val keys = (1L to 200L) val path = writeKeyOnlyParquetFile(dir, keys, rowGroupSize = 256L) val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsExcludedRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") val rowsExcludedPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") val bytesAvoidedRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") val bytesAvoidedPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") // k >= 195 -> last 6 keys survive; preceding row groups skipped or page-pruned. val filter = keyOnlyAtLeastFilter(Literal(195L), LongType, StorageFilterMetrics( rowGroupsSkipped = rgSkipped, + rowsExcludedByRowGroup = rowsExcludedRg, rowsExcludedWithinRowGroup = rowsExcludedPf, bytesAvoidedByRowGroup = bytesAvoidedRg, bytesAvoidedByPageFiltering = bytesAvoidedPf)) @@ -288,6 +260,13 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { assert(result.toSet == expected, s"expected exact survivor keys; got ${result.sorted}, expected ${expected.toSeq.sorted}") assert(rgSkipped.value >= 1, s"expected row groups skipped; got ${rgSkipped.value}") + assert(rowsExcludedRg.value > 0, + s"a skipped row group must count its rows too; got ${rowsExcludedRg.value}") + // The all-keys path takes its kept-row count from `finalRowCount` rather than from a + // phase-2 page store, so this identity is the only thing that checks that arithmetic. + assert(result.size + rowsExcludedRg.value + rowsExcludedPf.value == keys.size, + s"${result.size} emitted plus ${rowsExcludedRg.value} plus ${rowsExcludedPf.value} " + + s"should account for all ${keys.size} rows") // For an all-keys projection, the no-storage-filter path would have read the same key // column phase 1 reads. There are no non-key bytes to avoid; both metrics are 0. assert(bytesAvoidedRg.value == 0, @@ -439,7 +418,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("rewriteForMissingKeys: missing key with an existence DEFAULT probes the default's hash") { // A missing column that has a non-null existence DEFAULT is materialized by // ParquetColumnVector as that default, not as null. The predicate must therefore be evaluated - // against the default -- evaluating against null could skip a whole file whose rows all match. + // against the default. Evaluating against null could skip a whole file whose rows all match. val defaultValue = 7L val requested = StructType(Seq(StructField("k", LongType, nullable = true))) val bf = BloomFilter.create(10, 128) @@ -447,13 +426,15 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val expr = BloomFilterMightContain( Literal(bloomBytes(bf), BinaryType), new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))) - val filter = ParquetStorageFilter.create(Seq(expr), requested) - // Substituting the default keeps the file, because the default's hash is in the bloom. - assert(filter.rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(), + assert(ParquetStorageFilter.create(Seq(expr), requested) + .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(), "substituting the existence default must probe the default's hash and keep the file") - // Substituting null instead would drop it -- the bug this guards against. - assert(!filter.rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(), + // Substituting null instead would drop it, the bug this guards against. A second filter, + // because a rewrite is cached per set of missing positions: one scan always substitutes the + // same values for them. + assert(!ParquetStorageFilter.create(Seq(expr), requested) + .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(), "sanity check: substituting null probes a different hash and would drop the file") } @@ -564,7 +545,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // // Note the predicate deliberately uses `>` rather than `!=`. ColumnIndexFilter substitutes // `rangesForMissingColumns` for a predicate over a column outside its path set, and that is - // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq -- so a `!=` predicate here would be + // EMPTY for Gt/GtEq/Lt/LtEq/Eq but allRows for NotEq, so a `!=` predicate here would be // satisfied by a phase 0 that saw the wrong schema, and would prove nothing. withTempDir { dir => val rows = (1L to 200L).map(i => (i, f"v_$i%03d")) @@ -634,7 +615,9 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("FileSourceStrategy extracts bloom filter into scan.storageFilters when conf is on") { + test("FileSourceStrategy offers the bloom to the scan and keeps it post-scan too") { + // The scan gets the conjunct to prune with, and the post-scan Filter keeps it, so the answer + // never depends on what the reader managed to do with it. withBloomFilterTables { withSQLConf( SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", @@ -647,8 +630,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { assert(storageBlooms >= 1, s"expected >= 1 bloom filter on scan.storageFilters; got $storageBlooms.\n" + s"Plan:\n$plan") - assert(postScanBlooms == 0, - s"expected no bloom filter in any post-scan FilterExec; got $postScanBlooms.\n" + + assert(postScanBlooms >= 1, + s"expected the bloom to stay in a post-scan FilterExec; got $postScanBlooms.\n" + s"Plan:\n$plan") } } @@ -699,18 +682,6 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("ignoreCorruptFiles does not swallow a failure to honor a storage filter") { - // The reader has to fail when a file cannot support late materialization, because the planner - // removed the conjunct from the post-scan Filter. FileScanRDD and FilePartitionReader would - // skip the rest of such a file under ignoreCorruptFiles, silently dropping readable rows, so - // the exception the reader throws is excluded from that. - assert(!DataSourceUtils.shouldIgnoreCorruptFileException( - new UnsupportedFileReadException("cannot honor a storage filter"))) - // The generic reader failures it is carved out of stay swallowed. - assert(DataSourceUtils.shouldIgnoreCorruptFileException(new IllegalStateException("corrupt"))) - assert(DataSourceUtils.shouldIgnoreCorruptFileException(new java.io.IOException("truncated"))) - } - test("the feature still engages when ignoreCorruptFiles is on") { withBloomFilterTables { withSQLConf( @@ -723,10 +694,18 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val storageBlooms = countBloomFiltersInStorageFilters(plan) assert(storageBlooms == 1, s"expected the bloom on scan.storageFilters; got $storageBlooms.\nPlan:\n$plan") - assert(countBloomFiltersInPostScanFilters(plan) == 0, - s"expected no bloom left above the scan.\nPlan:\n$plan") + assert(countBloomFiltersInPostScanFilters(plan) >= 1, + s"and the post-scan Filter keeps it.\nPlan:\n$plan") assert(rows.map(r => (r.getLong(0), r.getLong(1))).toSet == Set((5L, 5L)), s"and the query must still return the joined row; got ${rows.mkString(", ")}") + // The three assertions above hold whether or not the reader honored the filter, since the + // post-scan Filter answers the query either way. The metrics are what say it did. + val scan = plan.collect { case s: FileSourceScanExec => s } + .find(_.storageFilters.nonEmpty).getOrElse(fail(s"no scan with storage filters:\n$plan")) + val excluded = + scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value + + scan.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value + assert(excluded > 0, s"the reader must have excluded rows; the metrics say $excluded") } } } @@ -803,7 +782,9 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } val (splicedRows, splicedBytes) = run("64MB") - val (plainRows, plainBytes) = run("1b") + // 100 bytes is past what 16 survivors of a long key buffer, and still leaves room for the + // row ranges: `k >= 350` keeps a contiguous run, so there is one range to hold. + val (plainRows, plainBytes) = run("100b") assert(splicedRows == 51, s"the filter keeps keys 350..400; got $splicedRows") assert(plainRows == splicedRows, s"rows differ past the cap: plain=$plainRows spliced=$splicedRows") @@ -814,6 +795,154 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } + test("row groups that splice and row groups over the cap, in both orders") { + // The emitted batch is one object for the whole read, so its key slots have to follow the path + // each row group took. This file makes all four cases occur in order, at 100 rows per row group + // and a batch capacity of 16: + // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row + // group splices; + // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected + // column. The key slots must go back to the persistent vectors here, or the batch reads keys + // out of a vector the previous row group's last batch already released; + // - rows 201-300: 6 survivors, splicing again, which needs the accumulators that giving + // splicing up released to be allocated afresh; + // - rows 301-400: no survivor at all, so the row group is skipped whole. + // + // Hence the assertion on the values rather than on the row count alone, and the skipped row + // group metric, which pins that the file really did split into several row groups. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val k = BoundReference(0, LongType, nullable = true) + def between(lo: Long, hi: Long): Expression = + And(GreaterThanOrEqual(k, Literal(lo)), LessThanOrEqual(k, Literal(hi))) + val filter = ParquetStorageFilter.create( + Seq(Or(Or(LessThanOrEqual(k, Literal(8L)), between(150L, 200L)), between(250L, 255L))), + StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))), + StorageFilterMetrics(rowGroupsSkipped = rgSkipped), + maxSplicedRowGroupBytes = 100L) + val (result, reader) = readAllWith(path, Seq("k", "v"), filter, + (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString), + capacity = 16) + try { + val expected = rows.filter { case (key, _) => + key <= 8L || (key >= 150L && key <= 200L) || (key >= 250L && key <= 255L) + } + assert(result == expected, + s"expected each surviving key with its own value; got ${result.take(20)}") + assert(rgSkipped.value >= 1, + s"expected a row group with no survivor at all; got ${rgSkipped.value}") + } finally { + reader.close() + } + } + } + + test("a row group past the cap charges its second key read against the byte metric") { + // Giving splicing up means phase 2 reads the key columns a second time, while the baseline + // counts them once, in phase 1. That extra read is a cost against the saving, so the same file + // and filter must report less avoided than they do while splicing. Without that term the metric + // would credit the feature with bytes it did transfer. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + val fileSchema = StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))) + + def avoidedBytes(cap: Long): Long = { + val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") + val filter = ParquetStorageFilter.create( + Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))), + fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap) + val (_, reader) = readAllWith(path, Seq("k", "v"), filter, + (b, i) => b.column(0).getLong(i), capacity = 16) + try pf.value finally reader.close() + } + + val spliced = avoidedBytes(64L * 1024 * 1024) + val plain = avoidedBytes(100L) + assert(spliced > 0, s"page filtering has to avoid something here; got $spliced") + assert(plain < spliced, + s"the second key read must count against the saving; plain=$plain spliced=$spliced") + } + } + + test("a row group whose survivors scatter into too many ranges is read without the filter") { + // `finalRanges` is built one row at a time, so an alternating filter makes one range per + // surviving row, and every column reader phase 2 drives materializes that list for the row + // group. Past the cap the filter is given up for that row group and every one of its rows is + // emitted, which the post-scan Filter then narrows. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) + val fileSchema = StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))) + val everyOtherRow = EqualTo( + Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L)) + + def rowsWithCap(cap: Long, capacity: Int = 4096): Int = { + val filter = + ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap) + val (emitted, reader) = + readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity) + try emitted.size finally reader.close() + } + + assert(rowsWithCap(64L * 1024 * 1024) == 200, + "with room for the ranges, only the surviving rows come back") + // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the + // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At + // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter + // up together: keeping one without the other would leave phase 2 reading non-key columns + // while emit spliced keys from a queue holding only the first survivors. + assert(rowsWithCap(1024L) == 400, + "past the cap after phase 1, the filter is given up and the row group comes back whole") + assert(rowsWithCap(1024L, capacity = 16) == 400, + "and the same past the cap inside phase 1's survivor loop") + } + } + + test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") { + // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a + // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the + // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate + // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or + // this query would start failing when the conf is turned on. + withTable("ansi_app", "ansi_build") { + // One file, one row group, with the non-numeric row inside it: a separate file would be + // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row. + spark.range(600) + .selectExpr( + "CASE WHEN id = 7 THEN 'not-a-number' ELSE CAST(id AS STRING) END AS k", + "CASE WHEN id = 7 THEN 'text' ELSE 'num' END AS kind") + .repartition(1).write.format("parquet").saveAsTable("ansi_app") + spark.range(30).selectExpr("id AS k", "id AS v") + .write.format("parquet").saveAsTable("ansi_build") + withSQLConf( + SQLConf.ANSI_ENABLED.key -> "true", + SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", + SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", + SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", + SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false") { + val query = "SELECT a.k FROM ansi_app a JOIN ansi_build b ON CAST(a.k AS BIGINT) = b.k " + + "WHERE b.v = 5 AND a.kind = 'num'" + val df = spark.sql(query) + val plan = df.queryExecution.executedPlan + assert(countBloomFiltersInPostScanFilters(plan) >= 1, + s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan") + assert(countBloomFiltersInStorageFilters(plan) == 0, + s"a cast key must not be extracted.\nPlan:\n$plan") + assert(df.collect().map(_.getString(0)).toSet == Set("5"), + "and the query must run rather than fail on the non-numeric row") + } + } + } + test("FileSourceStrategy leaves a non-deterministic bloom in the post-scan Filter") { // `ParquetStorageFilter.test` evaluates the predicate without calling // `BasePredicate.initialize(partitionIndex)`, which `GeneratePredicate` emits for a @@ -838,16 +967,17 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { countBloomFiltersInPostScanFilters(physical)) } - // Control: the same bloom over the key alone is extracted, so the arms differ in exactly - // one thing. + // Control: the same bloom over the key alone is offered to the scan, so the arms differ in + // exactly one thing. It stays in the post-scan Filter either way. val (deterministicInScan, deterministicPostScan) = extractedBlooms(new XxHash64(Seq(k))) - assert(deterministicInScan == 1 && deterministicPostScan == 0, - s"a deterministic bloom must be extracted; got scan=$deterministicInScan " + + assert(deterministicInScan == 1 && deterministicPostScan == 1, + s"a deterministic bloom must reach the scan; got scan=$deterministicInScan " + s"postScan=$deterministicPostScan") - // `Rand` contributes no reference, so `k` is still the only key column and only the - // determinism gate can reject this one. + // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject + // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside + // the format's answer, whose whitelist is deterministic expressions only. val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L)))) assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic") val (inScan, postScan) = extractedBlooms(nonDeterministic) @@ -866,21 +996,45 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { keyExpr: String, n: Long = 100L, rowGroupSize: Long = 256L, - dictionary: Boolean = false): String = { - val outDir = new File(dir, s"test-${System.nanoTime()}").getAbsolutePath - spark.range(1, n + 1).selectExpr(s"$keyExpr AS k") - .repartition(1) - .write - .option(ParquetOutputFormat.BLOCK_SIZE, rowGroupSize) - .option(ParquetOutputFormat.ENABLE_DICTIONARY, dictionary.toString) - .parquet(outDir) - val files = new File(outDir).listFiles((_, name) => name.endsWith(".parquet")) - assert(files != null && files.length == 1, s"expected exactly one parquet file under $outDir") - files(0).getAbsolutePath + dictionary: Boolean = false, + valueCopies: Int = 1): String = { + // Each of the `n` ids is written `valueCopies` times in a row, which is what makes parquet + // actually pick dictionary encoding when it is asked for: a dictionary writer falls back to + // PLAIN as soon as the dictionary plus the encoded ids is no smaller than the raw values, and + // all-distinct keys guarantee exactly that. The copies have to be consecutive, since the + // decision is per column chunk, and a row group holding one copy of each value is no better + // off than a row group of distinct ones. + // `div`, not `/`: `/` is floating-point division in Spark SQL, which would hand every key + // expression a double and quietly change the written type. + val ids = spark.range(0, n * valueCopies).selectExpr(s"(id div $valueCopies) + 1 AS id") + writeSingleParquetFile(dir, ids.selectExpr(s"$keyExpr AS k"), rowGroupSize, + dictionary = dictionary) + } + + // The encodings parquet actually used, over every column chunk of the file. Asserted rather than + // assumed, because asking for dictionary encoding does not mean getting it. + private def encodingsOf(filePath: String): Set[Encoding] = { + val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath)) + try { + reader.getFooter.getBlocks.asScala + .flatMap(_.getColumns.asScala) + .flatMap(_.getEncodings.asScala) + .toSet + } finally { + reader.close() + } } // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which - // selects the plain (non-splicing) vectorized path. + // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through + // here. + // + // `tryInitializeResource` closes the reader if anything inside throws and leaves it open + // otherwise, which is the contract these helpers need: the caller closes it once its assertions + // pass. Without it a failure in the read loop, which is what these tests are looking for, leaks + // the reader, its input stream and its off-heap vectors for the rest of the JVM, and can cascade + // into unrelated failures in the same suite. `initialize` throws too, so the wrap starts at + // construction. private def readAllWith[T]( filePath: String, columns: Seq[String], @@ -989,7 +1143,14 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // decode branch rather than straight out of the value array. for { (name, keyExpr, dt, threshold) <- keyTypeCases - dictionary <- Seq(false, true) + // Two types have no dictionary arm to take, and parquet's own writer factory says why: there is + // "no dictionary encoding for boolean", and for FIXED_LEN_BYTE_ARRAY, which is what a + // byte-array DECIMAL maps to, "dictionary encoding was not enabled in PARQUET 1.0", which is + // the writer version Spark writes by default. Asking for it yields PLAIN, so that arm would + // be the plain one over again. + dictionary <- + if (dt == BooleanType || DecimalType.isByteArrayDecimalType(dt)) Seq(false) + else Seq(false, true) } { val encoding = if (dictionary) "dictionary-encoded" else "plain-encoded" test(s"key type $name ($encoding): survivors round-trip through the phase 1 accumulators") { @@ -997,11 +1158,16 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // int96AsTimestamp and which is not the INT64 copier branch we want to cover here. withSQLConf(SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> "TIMESTAMP_MICROS") { withTempDir { dir => - val path = writeKeyOnlyParquetFileFromSql(dir, keyExpr, dictionary = dictionary) + // Three copies of every value on the dictionary arm, so the writer keeps the dictionary + // instead of falling back to PLAIN. + val path = writeKeyOnlyParquetFileFromSql( + dir, keyExpr, dictionary = dictionary, valueCopies = if (dictionary) 3 else 1) + assert(encodingsOf(path).exists(_.usesDictionary) == dictionary, + s"$name should be $encoding but parquet used ${encodingsOf(path).mkString(", ")}") val bound = GreaterThanOrEqual( BoundReference(0, dt, nullable = true), Literal.create(threshold, dt)) val expected = survivorsViaPlainPath(path, dt, bound) - assert(expected.nonEmpty && expected.size < 100, + assert(expected.nonEmpty && expected.size < 100 * (if (dictionary) 3 else 1), s"the $name case should keep some but not all rows; kept ${expected.size}") val requested = StructType(Seq(StructField("k", dt, nullable = true))) @@ -1097,14 +1263,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("two key columns: both accumulators stay aligned with each other and with the data column") { withTempDir { dir => - val outDir = new File(dir, "twokeys").getAbsolutePath - spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c") - .repartition(1) - .write - .option(ParquetOutputFormat.BLOCK_SIZE, 256L) - .option(ParquetOutputFormat.ENABLE_DICTIONARY, "false") - .parquet(outDir) - val path = new File(outDir).listFiles((_, n) => n.endsWith(".parquet"))(0).getAbsolutePath + val path = writeSingleParquetFile(dir, + spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L) // a >= 100 AND b <= 300 => a in [100, 150] val bound = And( @@ -1130,6 +1290,42 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } + test("key ordinals collected out of order still pair with the right batch slots") { + // `ParquetStorageFilter.create` sorts the ordinals it collects, and the emit path depends on + // that: it pairs the k-th key slot of the batch with key-row position k, which is the identity + // only while the list is ascending. Nothing else in the suite supplies an unsorted one, because + // every other fixture happens to mention its keys in column order. Production does not promise + // that: the conjuncts arrive in `afterScanFilters` order, which follows the join. Without the + // sort this returns `a` and `b` swapped, with no exception to notice. + withTempDir { dir => + val path = writeSingleParquetFile(dir, + spark.range(1, 201).selectExpr("id AS a", "id * 2 AS b", "CONCAT('v_', id) AS c"), 256L) + + // Same predicate as the test above, written so that `b` (ordinal 1) is collected first. + val bound = And( + LessThanOrEqual(BoundReference(1, LongType, nullable = true), Literal(300L)), + GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(100L))) + val requested = StructType(Seq( + StructField("a", LongType, nullable = true), + StructField("b", LongType, nullable = true), + StructField("c", StringType, nullable = true))) + val filter = ParquetStorageFilter.create(Seq(bound), requested) + assert(filter.keyColumnIndices.toSeq == Seq(0, 1), + s"key ordinals must come out ascending; got ${filter.keyColumnIndices.toSeq}") + + val (result, reader) = readAllWith(path, Seq("a", "b", "c"), filter, + (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i), + b.column(2).getUTF8String(i).toString)) + try { + val expected = (100L to 150L).map(i => (i, i * 2, s"v_$i")) + assert(result == expected, + s"expected a in [100,150] with b and c aligned; got ${result.take(5)} (${result.size})") + } finally { + reader.close() + } + } + } + test("partition columns are preserved alongside spliced key columns") { // Exercises the `i < isKeyTopLevel.length` branch of the emit loop: the partition slot sits // past the end of isKeyTopLevel and must come from persistentBatchColumns, not the key queues. @@ -1335,7 +1531,13 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // Attaches `storageFilters` to the scan of `SELECT id, k FROM

` and collects the result. private def collectWithStorageFilterOnKey( table: String, - buildFilter: Attribute => Expression): Set[(Long, Option[Long])] = { + buildFilter: Attribute => Expression): Set[(Long, Option[Long])] = + scanWithStorageFilterOnKey(table, buildFilter)._2 + + // As above, and also returns the scan, whose metrics the caller can then read. + private def scanWithStorageFilterOnKey( + table: String, + buildFilter: Attribute => Expression): (FileSourceScanExec, Set[(Long, Option[Long])]) = { val df = spark.sql(s"SELECT id, k FROM $table") val plan = df.queryExecution.executedPlan val scan = plan.collect { case s: FileSourceScanExec => s }.headOption @@ -1344,13 +1546,14 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val withSF = scan.copy(storageFilters = Seq(buildFilter(keyAttr))) val rowPlan = if (withSF.supportsColumnar) ColumnarToRowExec(withSF) else withSF // Executing the scan directly bypasses the Project that would reorder to the SELECT order, so - // rows arrive in the scan's own order -- the relation's dataSchema order, not the SELECT's. + // rows arrive in the scan's own order, the relation's dataSchema order and not the SELECT's. // Resolve positions by name rather than assuming they line up. val idPos = scan.output.indexWhere(_.name == "id") val kPos = scan.output.indexWhere(_.name == "k") - rowPlan.executeCollect() + val collected = rowPlan.executeCollect() .map(r => (r.getLong(idPos), if (r.isNullAt(kPos)) None else Some(r.getLong(kPos)))) .toSet + (withSF, collected) } test("missing key column with an existence DEFAULT is filtered on the default, not on null") { @@ -1379,11 +1582,22 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "true") { withEvolvedKeyTable("k BIGINT DEFAULT 7") { table => - val collected = collectWithStorageFilterOnKey( + val (scan, collected) = scanWithStorageFilterOnKey( table, k => GreaterThanOrEqual(k, Literal(30L))) val expected: Set[(Long, Option[Long])] = Set((4L, Some(40L)), (5L, Some(50L))) assert(collected == expected, s"got ${collected.toSeq.sorted}; expected ${expected.toSeq.sorted}") + // Rejecting a file whole is its own metric path, which walks every one of its row groups + // rather than going through the per-row-group loop. Nothing else asserts that walk, so it + // could report zero and only the rows above would notice. + def metric(name: String): Long = scan.metrics(name).value + assert(metric(FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED) >= 1, + "the older file's row groups count as skipped") + assert(metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP) == 3, + "and all three of its rows as excluded by a row group, got " + + metric(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP)) + assert(metric(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) > 0, + "and its projected bytes as avoided") } } } @@ -1488,6 +1702,13 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { Literal.create(null, BinaryType), XxHash64(Seq(AttributeReference("v", VariantType)()), 42L)) assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf") + // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the + // cast throws on a string that is not a number, and the reader evaluates the predicate on every + // row the pushed filter left, including those an earlier conjunct would have dropped. + val onCast = BloomFilterMightContain( + Literal.create(null, BinaryType), + XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L)) + assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded") // And the default is no support at all. assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom)) } @@ -1512,10 +1733,10 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("a scan with storage filters fails loudly if the vectorized reader is disabled later") { - // preparedStorageFilters deliberately does not re-check the conf, because by then the bloom is - // already gone from the post-scan Filter. So a vectorized-reader conf flipped between planning - // and execution must fail rather than quietly return every row. + test("a scan reads plainly when the vectorized reader is disabled after planning") { + // preparedStorageFilters deliberately does not re-check the conf. The reader then cannot honor + // the filter, and since the post-scan Filter keeps it, not honoring it is a slower read rather + // than a wrong one: every row of the file comes back from the scan. withTempDir { dir => val rows = (1L to 50L).map(i => (i, s"v_$i")) val path = writeParquetFile(dir, rows) @@ -1523,37 +1744,34 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { scanWithStorageFilter(path, "k", threshold = 25L) } withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") { - val e = intercept[Exception] { - executePlanCollect(withSF) - } - val message = Option(e.getCause).map(_.getMessage).getOrElse(e.getMessage) - assert(message != null && message.contains("Cannot honor storage filters"), - s"expected a clear storage-filter failure; got: $message") + val keys = executePlanCollect(withSF).map(_._1).toSet + assert(keys == (1L to 50L).toSet, + s"the filter is a hint, so an unfiltered read is expected; got ${keys.size} rows") } } } - test("a file format without storage-filter support rejects a non-empty storageFilters") { - // The default `FileFormat.buildReaderWithStorageFilters` body must not drop the filters it is - // handed: extraction has already removed them from the post-scan Filter, so a reader that - // ignores them returns rows the filter rejects. Only the planner's - // `getClass == classOf[ParquetFileFormat]` gate keeps this unreachable today, and that gate - // lives in another file. + test("a file format without storage-filter support ignores a non-empty storageFilters") { + // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed, + // because the post-scan Filter still holds it. Unreachable in production, since the planner + // asks `supportsStorageFilter` first. val storageFilters = Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L))) - val e = intercept[IllegalArgumentException] { + val e = intercept[Exception] { new NoStorageFilterFileFormat().buildReaderWithStorageFilters( spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters, Map.empty, new Configuration()) } - assert(e.getMessage.contains("does not support storage-filter pushdown"), e.getMessage) + // This stub implements no reader at all, so the error it fails with is the delegation's, which + // is what shows the filters were not rejected. + assert(e.getMessage.contains("buildReader is not supported"), e.getMessage) } Seq(false, true).foreach { aqe => test(s"FileSourceStrategy extraction preserves query results (AQE = $aqe)") { // AQE is on by default in production, and it is where the bloom subquery is planned by - // PlanAdaptiveSubqueries rather than PlanSubqueries -- the path preparedStorageFilters' - // ScalarSubquery materialization depends on. + // PlanAdaptiveSubqueries rather than PlanSubqueries, which is the path + // preparedStorageFilters' ScalarSubquery materialization depends on. withBloomFilterTables { val baseConf = Map( SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", @@ -1658,8 +1876,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("pushed data filter narrows to a page subset: phase 1 stays aligned with the row indexes") { // Every other fixture writes one page per column per row group, so column-index filtering can // only ever drop whole row groups and `pushedFilterRanges` is always the entire block. That - // makes the phase 1 alignment -- the r-th row readBatch delivers must pair with - // rowIndexIter.nextLong() -- hold trivially. With a small page size the data filter narrows + // makes the phase 1 alignment hold trivially, that the r-th row readBatch delivers pairs with + // rowIndexIter.nextLong(). With a small page size the data filter narrows // to a page subset, so the two sequences only agree if the pairing is actually correct. withTempDir { dir => val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) @@ -1715,8 +1933,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // Phase 0 asks parquet for row ranges only when column-index filtering is on, because // ParquetFileReader.getRowRanges checks whether a filter is pushed and NOT whether the user // enabled the column index. That branch is the escape hatch for a file whose column index is - // wrong -- trusting one here would drop rows for good, since the post-scan Filter no longer - // holds the predicate -- and nothing exercised it. + // wrong, and nothing exercised it. Trusting a wrong one drops rows for good: every phase reads + // within phase 0's ranges, and no filter above the scan can bring back a row it never read. // // The row accounting is what tells the two arms apart. Everything is scoped to the rows the // pushed data filter left, so with the column index on, the rows it prunes at page level never @@ -1872,35 +2090,6 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("all-key projection: byte metrics are zero and no offset index work is needed") { - // With every projected column a key, phase 2 never runs, so baseline == phase1 and both byte - // metrics are 0 by construction. This is the shape the design notes call the biggest win, so it - // must not be the shape that pays for metrics. - withTempDir { dir => - val path = writeKeyOnlyParquetFileFromSql(dir, "id", n = 200L) - val bytesRg = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByRg") - val bytesPf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") - val rowsRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") - val bound = GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(190L)) - val requested = StructType(Seq(StructField("k", LongType, nullable = true))) - val filter = ParquetStorageFilter.create(Seq(bound), requested, StorageFilterMetrics( - rowsExcludedByRowGroup = rowsRg, - bytesAvoidedByRowGroup = bytesRg, - bytesAvoidedByPageFiltering = bytesPf)) - val (result, reader) = - readAllWith(path, Seq("k"), filter, (b, i) => b.column(0).getLong(i)) - try { - assert(result == (190L to 200L), s"expected keys 190..200; got $result") - assert(bytesRg.value == 0 && bytesPf.value == 0, - s"an all-key projection has no non-key bytes to avoid; got rg=${bytesRg.value} " + - s"pf=${bytesPf.value}") - assert(rowsRg.value > 0, "row groups should still be skipped, and counted in rows") - } finally { - reader.close() - } - } - } - // ----- Projection order and batch boundaries ----- test("non-key column before the key column: emit maps queues to the right batch slots") { @@ -1980,7 +2169,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // WholeStageCodegenExec, and only when the row loop exits with a batch still in hand. That exit // is the limit check, which needs a limit inside the same codegen stage. So the plan is built // with LocalLimitExec and handed to CollapseCodegenStages, and the generated source is asserted - // to contain the close -- without that, this test would pass for the wrong reason. + // to contain the close. Without that, this test would pass for the wrong reason. // // What it exercises: the spliced batch's columns are closed from outside while the reader is // still open, and the reader's own close() then runs over the same vectors. @@ -2034,7 +2223,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val a = scan.output.find(_.name == "a").getOrElse(fail("no a")) val b = scan.output.find(_.name == "b").getOrElse(fail("no b")) // Two key columns. In the older file `b` is missing and reads as its default 7, so the - // predicate must be evaluated with 7 substituted for it -- and `a >= 2` still filters. + // predicate must be evaluated with 7 substituted for it, and `a >= 2` still filters. val withSF = scan.copy(storageFilters = Seq( GreaterThanOrEqual(a, Literal(2L)), GreaterThanOrEqual(b, Literal(5L)))) @@ -2063,7 +2252,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { /** * A [[FileFormat]] that does not override `buildReaderWithStorageFilters`, so it exercises the - * default body's rejection of storage filters it cannot honor. + * default body, which ignores storage filters it cannot honor. */ private class NoStorageFilterFileFormat extends FileFormat { override def inferSchema( From c29625f54c67403caeda96630892f31f4f4e9fd5 Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Fri, 25 Sep 2026 15:54:49 +0200 Subject: [PATCH 4/6] Review fixes (round 3) dongjoon-hyun's third round and cloud-fan's review. - Fail open instead of declining a conjunct the reader might not be able to evaluate on every row: an error that carries an error class and is not internal gives the filter up for that row group, and the post-scan Filter then evaluates the conjuncts in their own order. Requiring a total expression also declined every widening cast, which is what type coercion inserts for an int-to-bigint join - Coalesce row ranges lazily in ParquetReadState, and hold the current range as two longs, so one range at a time is held per column reader instead of a list each. That is what made the memory cap reachable, and every column-index read is lighter for it - Weigh that cap after every surviving row, over the buffered key bytes and the row ranges together, so a row group whose survivors fit in one accumulator and a range list that grows after the buffer is gone are both bounded. The check after phase 1 goes with it, the loop catches everything now - Keep applying the filter to a file with no offset index wherever it empties a row group, rather than giving the whole file up - Close the page stores phase 1 and phase 2 read from - Pair key slots with queues through keyColumnIndices, so the emit path does not depend on the ordinals being sorted - Ask the format once per scan whether it applies storage filters at all, which is where the conf that enables them belongs, and offer the original conjuncts rather than their canonicalized form - Enforce the column-index-store ordering the byte metrics rely on, with the footer rather than a comment - Cover what nothing asserted: a file with no offset index, an unsorted ordinal collection, the cap crossed inside the survivor loop, the metrics of a whole-file skip, and that AQE keeps the filter on the scan - Replace three metric assertions that could not fail, and correct the stale names and comments two reviewers found --- .../apache/spark/sql/internal/SQLConf.scala | 36 +- .../datasources/parquet/ParquetReadState.java | 118 ++--- .../VectorizedParquetRecordReader.java | 396 ++++++++++------- .../sql/execution/DataSourceScanExec.scala | 33 +- .../execution/datasources/FileFormat.scala | 30 +- .../datasources/FileSourceStrategy.scala | 14 +- .../parquet/ParquetFileFormat.scala | 23 +- .../parquet/ParquetStorageFilter.scala | 58 ++- .../parquet/ParquetStorageFilterSuite.scala | 411 +++++++++++++----- 9 files changed, 711 insertions(+), 408 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index a0d64ff65b5b2..06646331de02f 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1956,11 +1956,12 @@ object SQLConf { "false, no storage filter is attached to a scan in the first place and the filter is " + "applied as an ordinary post-scan filter alone. A pushed filter stays in the post-scan " + "filter as well, the way a pushed data filter does, so honoring it is optional: a reader " + - "that meets a file it cannot prune, one written without a Parquet page index say, reads " + - "it the way a plain scan would. That costs the filter's own evaluation, plus one " + - "key-column read of the first row group the reader tries it on, since a missing page " + - "index is only reported by attempting the read. Note that the surviving key values of a " + - "whole row group are buffered before the " + + "that meets a row group it cannot prune reads it the way a plain scan would, plus one " + + "more read of the key columns, since the phase that evaluated the filter already read " + + "them. A file written with no Parquet page index is read with the filter applied only " + + "where it empties a whole row group, since narrowing to part of one needs that index, so " + + "every row group of it that keeps a row pays that. Note that " + + "the surviving key values of a whole row group are buffered before the " + "first batch of that row group is produced, so a task holds up to one extra copy of the " + "key columns for one row group.") .version("5.0.0") @@ -1971,19 +1972,18 @@ object SQLConf { val PARQUET_STORAGE_FILTER_PUSHDOWN_MAX_SPLICED_ROW_GROUP_BYTES = buildConf("spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes") .internal() - .doc("Largest key-column buffer, in bytes, that the vectorized Parquet reader will hold to " + - "splice surviving key values into its output batches. Splicing buffers one key value per " + - "surviving row of a row group, so the reader counts what it has buffered and gives that " + - "row group up once the count passes this, reading every projected column of the " + - "surviving rows in one go instead, which costs one extra read of the key columns. What " + - "is counted is the buffered values and their per-row overhead, not the backing arrays, " + - "which a column vector may grow beyond that. The count is examined whenever a batch " + - "worth of survivors per key column has been buffered, so a row group whose survivors fit " + - "in a single batch is never weighed at all: it holds no more than the plain read path " + - "does. The same limit covers the row ranges the surviving rows fall into, which every " + - "column reader of the second phase holds a copy of, so the two share one budget. A row " + - "group whose ranges alone pass it is read without the filter applied at all, which is " + - "correct but as slow as not pushing the filter.") + .doc("Most memory, in bytes, that the vectorized Parquet reader holds for one row group " + + "while it applies a storage filter. Two things count against it, they sit in different " + + "pools, and both grow with the number of surviving rows: the key values buffered to " + + "splice into the output batches, which follow the reader's memory mode and so can be off " + + "heap, and the row ranges those rows fall into, always on heap, which the second phase " + + "needs to select its pages. The count is examined after every surviving row. Past the " + + "limit the reader releases the buffer and reads every projected column of the surviving " + + "rows instead, " + + "which costs one extra read of the key columns, and past it again it reads the row group " + + "with no filter applied at all, which is correct but as slow as not pushing the filter. " + + "What is counted is the buffered values and their per-row overhead, not the backing " + + "arrays, which a column vector may grow beyond that.") .version("5.0.0") .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE) .bytesConf(ByteUnit.BYTE) diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java index e29a16a834397..94320402436b7 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java @@ -19,30 +19,29 @@ import org.apache.parquet.column.ColumnDescriptor; -import java.util.ArrayList; -import java.util.Iterator; -import java.util.List; import java.util.PrimitiveIterator; /** * Helper class to store intermediate state while reading a Parquet column chunk. */ final class ParquetReadState { - /** A special row range used when there is no row indexes (hence all rows must be included) */ - private static final RowRange MAX_ROW_RANGE = new RowRange(Long.MIN_VALUE, Long.MAX_VALUE); + /** The row indexes to include, only not-null if the column index is present. */ + private final PrimitiveIterator.OfLong rowIndexes; /** - * A special row range used when the row indexes are present AND all the row ranges have been - * processed. This serves as a sentinel at the end indicating that all rows come after the last - * row range should be skipped. + * The current row range, as its bounds rather than as an object: one range per surviving row is + * what a filter with scattered survivors produces, for every column reader of every row group. + * + *

With no row indexes they are the whole range, since every row must be included. Once the + * indexes are exhausted they are inverted, which says that every row from there on is to be + * skipped. */ - private static final RowRange END_ROW_RANGE = new RowRange(Long.MAX_VALUE, Long.MIN_VALUE); + private long currentRangeStart; + private long currentRangeEnd; - /** Iterator over all row ranges, only not-null if column index is present */ - private final Iterator rowRanges; - - /** The current row range */ - private RowRange currentRange; + /** The row index that ended the current range by not continuing it, so it starts the next one. */ + private long pendingRowIndex; + private boolean hasPendingRowIndex; /** Maximum repetition level for the Parquet column */ final int maxRepetitionLevel; @@ -90,43 +89,10 @@ final class ParquetReadState { this.maxRepetitionLevel = descriptor.getMaxRepetitionLevel(); this.maxDefinitionLevel = descriptor.getMaxDefinitionLevel(); this.isRequired = isRequired; - this.rowRanges = constructRanges(rowIndexes); + this.rowIndexes = rowIndexes; nextRange(); } - /** - * Construct a list of row ranges from the given `rowIndexes`. For example, suppose the - * `rowIndexes` are `[0, 1, 2, 4, 5, 7, 8, 9]`, it will be converted into 3 row ranges: - * `[0-2], [4-5], [7-9]`. - */ - private Iterator constructRanges(PrimitiveIterator.OfLong rowIndexes) { - if (rowIndexes == null) { - return null; - } - - List rowRanges = new ArrayList<>(); - long currentStart = Long.MIN_VALUE; - long previous = Long.MIN_VALUE; - - while (rowIndexes.hasNext()) { - long idx = rowIndexes.nextLong(); - if (currentStart == Long.MIN_VALUE) { - currentStart = idx; - } else if (previous + 1 != idx) { - RowRange range = new RowRange(currentStart, previous); - rowRanges.add(range); - currentStart = idx; - } - previous = idx; - } - - if (previous != Long.MIN_VALUE) { - rowRanges.add(new RowRange(currentStart, previous)); - } - - return rowRanges.iterator(); - } - /** * Must be called at the beginning of reading a new batch. */ @@ -151,39 +117,51 @@ void resetForNewPage(int totalValuesInPage, long pageFirstRowIndex) { * Returns the start index of the current row range. */ long currentRangeStart() { - return currentRange.start; + return currentRangeStart; } /** * Returns the end index of the current row range. */ long currentRangeEnd() { - return currentRange.end; + return currentRangeEnd; } /** - * Advance to the next range. + * Advances to the next range, coalescing the run of ascending row indexes that forms it. For + * example `[0, 1, 2, 4, 5, 7, 8, 9]` yields `[0-2]`, then `[4-5]`, then `[7-9]`. + * + *

One range at a time on purpose. They are consumed once, in order, so holding them all buys + * nothing and costs a list per column reader of the row group, which for scattered survivors is + * one entry per row in every one of those lists. */ void nextRange() { - if (rowRanges == null) { - currentRange = MAX_ROW_RANGE; - } else if (!rowRanges.hasNext()) { - currentRange = END_ROW_RANGE; - } else { - currentRange = rowRanges.next(); + if (rowIndexes == null) { + currentRangeStart = Long.MIN_VALUE; + currentRangeEnd = Long.MAX_VALUE; + return; } + if (!hasPendingRowIndex && !rowIndexes.hasNext()) { + currentRangeStart = Long.MAX_VALUE; + currentRangeEnd = Long.MIN_VALUE; + return; + } + long start = hasPendingRowIndex ? pendingRowIndex : rowIndexes.nextLong(); + hasPendingRowIndex = false; + long end = start; + // A range can only be closed by seeing the index that does not continue it, so that index is + // held back for the next call. + while (rowIndexes.hasNext()) { + long idx = rowIndexes.nextLong(); + if (idx == end + 1) { + end = idx; + } else { + pendingRowIndex = idx; + hasPendingRowIndex = true; + break; + } + } + currentRangeStart = start; + currentRangeEnd = end; } - - /** - * Helper struct to represent a range of row indexes `[start, end]`. - */ - private record RowRange(long start, long end) { - } - - /** - * What one {@link RowRange} costs on the heap, for a caller that has to budget for the list this - * class builds: two longs, their object header, and the slot in the list holding them. It lives - * here because {@link RowRange} is private, so a caller cannot measure it. - */ - static final int ESTIMATED_ROW_RANGE_BYTES = 40; } diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index 059f8beb27502..679da1cffc2f8 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -210,23 +210,34 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa private int keyFixedBytesPerRow; /** Which key columns hold their values out of line, so a length has to be measured per row. */ private boolean[] keyVariableLength; - /** Key-value bytes buffered for the row group currently loading. */ - private long splicedBytes; - /** Ranges the surviving rows of the row group currently loading fall into. */ - private long survivorRangeCount; /** Whether the row group currently loading is read without the filter applied at all. */ private boolean filterGivenUp; /** - * Whether this file has no Parquet offset index for some projected column, which parquet only - * reports by throwing when a read asks for part of a block. It is a property of the file, so it - * is learned once: later row groups skip phase 1 rather than evaluate a filter they cannot use. + * The pages phase 2 read for the row group being emitted. Held because the column readers draw + * from it for the whole row group, and closed when the next one is loaded: `readFilteredRowGroup` + * hands out a store the file reader does not track, unlike `readNextRowGroup`. + */ + private PageReadStore dataPages; + + /** Whether the filter has already been reported as failing to evaluate on this file. */ + private boolean loggedFilterEvaluationError; + + /** Whether the byte metric has already been reported as undercounting on this file. */ + private boolean loggedMissingStoreEntry; + + /** + * Whether this file lacks a Parquet offset index for some column phase 2 needs, which is what it + * needs to read part of a row group. Learned from the read that fails, once, and then used to + * stop the reader from buffering key values phase 2 will have to read again. Phase 1 still runs + * on such a file: a row group the filter empties is skipped whole, which needs no index at all. */ private boolean fileHasNoOffsetIndex; + /** - * Whether the row group currently loaded is spliced. It starts true unless the file is already - * known to have no offset index, and turns false in phase 1 once the survivors buffered pass the - * cap. False means phase 2 read every projected column, key columns included, so the emit path - * takes them straight from the persistent batch. + * Whether the row group currently loaded is spliced. It starts true unless phase 2 will have to + * read every projected column anyway, and turns false in phase 1 once the survivors buffered pass + * the cap. False means phase 2 read every projected column, key columns included, so the emit + * path takes them straight from the persistent batch. */ private boolean spliceCurrentRowGroup; private int nextBlockIndex; @@ -379,9 +390,13 @@ public void close() throws IOException { keyScratchVectors = null; keyScratchBatch = null; } finally { - // lateMatReader aliases the base-class reader; super.close() owns it. - lateMatReader = null; - super.close(); + try { + closeDataPages(); + } finally { + // lateMatReader aliases the base-class reader; super.close() owns it. + lateMatReader = null; + super.close(); + } } } } @@ -649,15 +664,12 @@ private void publishSurvivorKeyVectors(int num) { } } keyVectorsPublished = true; - // Key slots are filled in ascending slot order while `keyIdx` walks the queues in key-list - // order, so the pairing is the identity only because `ParquetStorageFilter.create` sorts - // `keyColumnIndices` ascending. `isKeyTopLevel` says which slots are keys, not where each - // sits in that list, so this loop cannot re-derive the pairing: an unsorted list would swap - // key columns in the output batch. Only key slots are touched, since a non-key slot never - // holds anything but its persistent vector. - int keyIdx = 0; - for (int i = 0; i < isKeyTopLevel.length; i++) { - if (isKeyTopLevel[i]) spliceBatchColumns[i] = keyVectorQueues[keyIdx++].peekFirst(); + // Queue k holds the survivors of key column k, and `keyColumnIndices[k]` is the batch slot that + // key column sits in, so the pairing is read off rather than re-derived from slot order. Only + // key slots are touched, since a non-key slot never holds anything but its persistent vector. + int[] keyIndices = storageFilter.keyColumnIndices(); + for (int k = 0; k < keyIndices.length; k++) { + spliceBatchColumns[keyIndices[k]] = keyVectorQueues[k].peekFirst(); } } @@ -760,11 +772,14 @@ private void initializeLateMaterialization() throws IOException { storageFilter = storageFilter.rewriteForMissingKeys(missing, missingValues); if (presentKeyColumns.isEmpty()) { - // Every key column is missing, so the rewritten predicate is constant for this file. - boolean keepAll = storageFilter.evalAllMissing(); - if (!keepAll) recordFileSkipped(); + // Every key column is missing, so the rewritten predicate is constant for this file. An + // empty answer means evaluating it raised an error, and then the file is read the way a + // plain scan would read it rather than skipped. + Option keepAll = storageFilter.evalAllMissing(); + boolean skipFile = keepAll.isDefined() && !((boolean) keepAll.get()); + if (skipFile) recordFileSkipped(); storageFilter = null; - hitEndOfData = !keepAll; + hitEndOfData = skipFile; return; } } @@ -843,15 +858,14 @@ private void initializeSplicingState(List presentKeyColumns) { } /** - * What phase 2 of the current row group will hold for its row ranges. Every column reader it - * drives materializes the row group's range list of its own ({@code ParquetReadState}), and a - * filter whose survivors are scattered makes one range per surviving row. + * What the surviving rows of the current row group cost to hold as row ranges. A filter whose + * survivors are scattered makes one range per surviving row, and phase 2 needs the whole set to + * select its pages. One set is held, not one per column reader, because + * {@code ParquetReadState} coalesces the ranges it walks lazily. */ - private long rowRangeStateBytes(long rangeCount) { - int leaves = spliceCurrentRowGroup - ? (nonKeyColumns == null ? 0 : nonKeyColumns.size()) - : requestedColumns.size(); - return rangeCount * ParquetReadState.ESTIMATED_ROW_RANGE_BYTES * leaves; + private static long rowRangeStateBytes(long rangeCount) { + // Parquet's own `RowRanges.Range`: two longs, their object header, and the list slot for it. + return rangeCount * 40L; } /** Whether a key value lives in the vector's byte child rather than in its fixed-width array. */ @@ -947,6 +961,10 @@ private void checkEndOfRowGroup() throws IOException { * Sets {@link #hitEndOfData} when all row groups have been processed. */ private void loadNextRowGroupWithLateMaterialization() throws IOException { + // The previous row group is fully emitted by the time this is called, so its pages are done + // with. Released here rather than at the next assignment, so an all-keys row group, which reads + // no data pages at all, does not keep the one before it alive. + closeDataPages(); while (nextBlockIndex < totalBlockCount) { int blockIdx = nextBlockIndex++; long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount(); @@ -959,24 +977,17 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // the first batch, and that buffer is outside any MemoryConsumer, so phase 1 counts what it // holds against `maxSplicedRowGroupBytes` together with the row ranges phase 2 will hold. // Past that it gives splicing up, and past it again the filter itself, which is what - // `filterGivenUp` says. A file already known to have no offset index starts there. - filterGivenUp = fileHasNoOffsetIndex; - spliceCurrentRowGroup = !filterGivenUp; - splicedBytes = 0L; + // `filterGivenUp` says. + filterGivenUp = false; + // Nothing is buffered for a file phase 2 cannot read in part: it will read the key columns + // again along with everything else. + spliceCurrentRowGroup = !fileHasNoOffsetIndex; // Phase 0: rows allowed by the pushed data filter, at column-index granularity. The full // requestedSchema goes back on first, because phases 1 and 2 narrow it and // ParquetFileReader.getRowRanges computes ranges against the reader's current paths. lateMatReader.setRequestedSchema(requestedColumns); - // getRowRanges checks only whether a filter is pushed, not options.useColumnIndexFilter(), - // so calling it unconditionally would keep applying column-index filtering after a user - // turned it off, which is the escape hatch for a file whose column index is wrong. Every - // phase below reads within these ranges, so a wrong column index would cost rows the plain - // path would have returned. Phase 2 is unaffected: it selects pages through the offset index, - // a separate structure this conf says nothing about. - RowRanges pushedFilterRanges = useColumnIndexFilter - ? lateMatReader.getRowRanges(blockIdx) - : RowRanges.createSingle(blockRowCount); + RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount); // RowRanges.rowCount() walks every range, so resolve each range set's count once. long baselineRows = pushedFilterRanges.rowCount(); if (baselineRows == 0) { @@ -990,53 +1001,57 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // a plain read of this projection would transfer for every row the pushed filter kept. The // null checks only skip work for a caller that drives this reader without a scan's metrics; // FileSourceScanLike creates all five whenever storageFilters is non-empty. - // compressedBytesForRowRanges never does IO of its own. A row group whose filter is already - // given up reports nothing either way, so it does not pay for the baseline at all. + // compressedBytesForRowRanges never does IO of its own. StorageFilterMetrics m = storageFilter.metrics(); SQLMetric bytesAvoidedRg = m.bytesAvoidedByRowGroup(); SQLMetric bytesAvoidedPf = m.bytesAvoidedByPageFiltering(); - boolean needBytes = (bytesAvoidedRg != null || bytesAvoidedPf != null) && !filterGivenUp; + boolean needBytes = bytesAvoidedRg != null || bytesAvoidedPf != null; Map blockChunks = needBytes ? chunksByPath(lateMatReader, blockIdx) : null; long nonKeyBaselineBytes = needBytes - ? compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, nonKeyColumns, + ? compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns, pushedFilterRanges, baselineRows) : 0L; // Phase 1: switch to key-only schema, read key columns under pushedFilterRanges, evaluate the - // storage filter per row. Skipped for a row group the filter is already given up for, which - // leaves every row of `pushedFilterRanges` to emit, exactly what a plain read would. + // storage filter per row. The defaults below are what a row group whose filter is given up + // emits, which is every row of `pushedFilterRanges`, exactly what a plain read would. RowRanges finalRanges = pushedFilterRanges; long finalRowCount = baselineRows; - if (!filterGivenUp) { - lateMatReader.setRequestedSchema(keyOnlyColumns); - PageReadStore keyPages = lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges); + lateMatReader.setRequestedSchema(keyOnlyColumns); + // Closed at the end of the phase that reads it. `readFilteredRowGroup` hands out a store + // the file reader does not track, unlike `readNextRowGroup`, so nothing else would. + RowRanges survivors; + try (PageReadStore keyPages = + lateMatReader.readFilteredRowGroup(blockIdx, pushedFilterRanges)) { if (keyPages == null) { - // Unreachable: readFilteredRowGroup returns null only for an empty block, and we already - // know pushedFilterRanges selects at least one row. Skipping the block here would drop - // its surviving rows from the output, so assert rather than `continue`. + // Unreachable: readFilteredRowGroup returns null only for an empty block, and we know + // pushedFilterRanges selects at least one row. Skipping the block here would drop its + // surviving rows from the output, so assert rather than `continue`. throw new IllegalStateException( "No key pages for row group " + blockIdx + " despite " + baselineRows + " rows selected by the pushed filter"); } - RowRanges survivors = evaluateStorageFilter(keyPages, pushedFilterRanges); - if (!filterGivenUp - && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupBytes()) { - // Phase 1 weighs the budget once per accumulator, so a row group whose survivors fit in - // a single one is only caught here, with its survivors buffered. Those are released, - // since the ranges they were spliced against are about to be thrown away. - giveUpFilter(); + survivors = evaluateStorageFilter(keyPages, pushedFilterRanges); + } + if (survivors != null) { + finalRanges = survivors; + finalRowCount = survivors.rowCount(); + if (finalRowCount == 0) { + // Every surviving row was rejected by the storage filter; skip the block entirely, + // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns, + // and that cost is not part of the baseline, so nothing is subtracted from it here. + recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes); + continue; } - if (!filterGivenUp) { - finalRanges = survivors; - finalRowCount = survivors.rowCount(); - if (finalRowCount == 0) { - // Every surviving row was rejected by the storage filter; skip the block entirely, - // which avoids the whole non-key baseline. Phase 1 still paid to read the key columns, - // and that cost is not part of the baseline, so nothing is subtracted from it here. - recordRowGroupSkipped(m, baselineRows, nonKeyBaselineBytes); - continue; - } + // Reading part of a row group needs the offset index, so on a file without one phase 2 + // reads the whole row group instead. The rows the filter rejected are then emitted and + // the post-scan Filter drops them. What the filter still saves on such a file is the row + // groups it empties, which is decided above and needs no index at all. + if (fileHasNoOffsetIndex && finalRowCount < baselineRows) { + giveUpFilter(); + finalRanges = pushedFilterRanges; + finalRowCount = baselineRows; } } @@ -1045,11 +1060,8 @@ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupByte // projection is all keys and their values were buffered, since emit then builds every batch // from the key queues alone. long keptRows; - long phase2Bytes; - PageReadStore dataPages = null; if (nonKeyColumns == null && spliceCurrentRowGroup) { keptRows = finalRowCount; - phase2Bytes = 0L; } else { lateMatReader.setRequestedSchema( spliceCurrentRowGroup ? nonKeyColumns : requestedColumns); @@ -1063,19 +1075,32 @@ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupByte // wall: a store missing one column's offset index reports no column index either, so // `getRowRanges` could not have narrowed anything and the ranges cover the whole block. // - // Nothing is checked up front, so a file with no page index still reads with the filter - // applied wherever the filter keeps a row group whole (`readFilteredRowGroup` degrades to a - // plain read when the ranges cover the block) or rejects one whole. + // Asked this way rather than up front, from the footer. Parquet resolves the index over the + // paths current at its first lookup for the block, which without a pushed data filter is + // the non-key columns alone, so a footer walk over the projection is both stricter than the + // read and blind to an index that is claimed but unreadable. It is also where the throw + // costs least: it lands before any data page is read. try { dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); } catch (MissingOffsetIndexException e) { - LOG.warn("Not applying the storage filter to {}: reading part of a row group needs a " - + "Parquet offset index, and this file was written without a page index for at least " - + "one projected column", e, MDC.of(LogKeys.PATH, lateMatReader.getFile())); + LOG.warn("Reading {} without page-level storage filtering: reading part of a row group " + + "needs a Parquet offset index, and this file has none for at least one column the " + + "read needs. Row groups the filter empties are still skipped whole", e, + MDC.of(LogKeys.PATH, lateMatReader.getFile())); fileHasNoOffsetIndex = true; giveUpFilter(); finalRanges = pushedFilterRanges; finalRowCount = baselineRows; + // The retry only avoids the same wall because a store missing one column's offset index + // reports no column index either, so these ranges cover the whole block and parquet reads + // it without consulting an index. That is three parquet internals deep, so it is checked: + // a release that changes any of them should fail here rather than throw from the read. + if (baselineRows != blockRowCount) { + throw new IllegalStateException(String.format( + "Cannot read row group %d of %s without an offset index: the pushed filter selects " + + "%d of %d rows, so a plain read of the block is not what it asks for", + blockIdx, lateMatReader.getFile(), baselineRows, blockRowCount)); + } lateMatReader.setRequestedSchema(requestedColumns); dataPages = lateMatReader.readFilteredRowGroup(blockIdx, finalRanges); } @@ -1087,30 +1112,25 @@ && rowRangeStateBytes(survivorRangeCount) > storageFilter.maxSplicedRowGroupByte + " rows to read"); } keptRows = dataPages.getRowCount(); - // Nothing is computed for a row group whose filter was given up: it read what a plain scan - // reads, so the answer is a certain zero. `needBytes`, not just `bytesAvoidedPf != null`, - // because that is what built `blockChunks`. - if (needBytes && bytesAvoidedPf != null && !filterGivenUp) { - phase2Bytes = compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, + // Nothing is reported for a row group whose filter was given up: it read what a plain scan + // reads, so the saving is a certain zero. + if (bytesAvoidedPf != null && !filterGivenUp) { + long phase2Bytes = compressedBytesForRowRanges(blockIdx, blockChunks, nonKeyColumns, finalRanges, finalRowCount); if (!spliceCurrentRowGroup) { // This row group gave splicing up, so phase 2 read the key columns a second time. The // baseline counts them once, in phase 1, so the extra read is a cost against it. - phase2Bytes += compressedBytesForRowRanges(lateMatReader, blockIdx, blockChunks, + phase2Bytes += compressedBytesForRowRanges(blockIdx, blockChunks, keyOnlyColumns, finalRanges, finalRowCount); } - } else { - phase2Bytes = 0L; + // `SQLMetric.add` ignores a negative value, so a row group that read more than the + // baseline after giving splicing up contributes nothing rather than subtracting. + bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes); } } long filteredRows = baselineRows - keptRows; SQLMetric rowsExcludedWithinRg = m.rowsExcludedWithinRowGroup(); if (rowsExcludedWithinRg != null && filteredRows > 0) rowsExcludedWithinRg.add(filteredRows); - if (bytesAvoidedPf != null && !filterGivenUp) { - // `SQLMetric.add` ignores a negative value, so a row group that read more than the baseline - // after giving splicing up contributes nothing rather than subtracting. - bytesAvoidedPf.add(nonKeyBaselineBytes - phase2Bytes); - } if (dataPages != null) { if (rowIndexGenerator != null) { @@ -1157,27 +1177,38 @@ private void recordFileSkipped() { for (int blockIdx = 0; blockIdx < blocks.size(); blockIdx++) { // Measured against the rows the pushed data filter kept, which is the baseline every other // skip path uses: the rows its column index already excluded were never this filter's to - // save. `getRowRanges` is a cache hit whenever the two can differ, because - // `getFilteredRecordCount()` at initialize resolved every block's ranges then. It has to be - // guarded the same way phase 0 guards it, since it consults the pushed filter but not the - // conf that turns column-index filtering off. + // save. Resolving them again is a cache hit whenever the two can differ, because + // `getFilteredRecordCount()` at initialize resolved every block's ranges then. long blockRowCount = blocks.get(blockIdx).getRowCount(); if (blockRowCount == 0) continue; - RowRanges blockRanges = useColumnIndexFilter - ? lateMatReader.getRowRanges(blockIdx) - : RowRanges.createSingle(blockRowCount); + RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount); long survivingRows = blockRanges.rowCount(); if (survivingRows == 0) continue; // The key columns are missing from this file, so they contribute nothing to the walk, and the // whole projection is what a plain read would have transferred. long avoidedBytes = needBytes - ? compressedBytesForRowRanges(lateMatReader, blockIdx, + ? compressedBytesForRowRanges(blockIdx, chunksByPath(lateMatReader, blockIdx), projected, blockRanges, survivingRows) : 0L; recordRowGroupSkipped(m, survivingRows, avoidedBytes); } } + /** + * The rows of a block the pushed data filter allows, at column-index granularity. + * + * {@code getRowRanges} checks only whether a filter is pushed, not + * {@code options.useColumnIndexFilter()}, so calling it unconditionally would keep applying + * column-index filtering after a user turned it off, which is the escape hatch for a file whose + * column index is wrong. Every phase reads within these ranges, so a wrong column index would + * cost rows a plain read would have returned. + */ + private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) { + return useColumnIndexFilter + ? lateMatReader.getRowRanges(blockIdx) + : RowRanges.createSingle(blockRowCount); + } + /** * The block's column chunks by path, built once per row group and shared by the byte-metric calls * that consume it, since {@link BlockMetaData} offers no lookup of its own. @@ -1214,8 +1245,7 @@ private static Map chunksByPath( *

Columns absent from this physical file (schema evolution) contribute nothing, which is * correct: the reader transfers nothing for them. */ - private static long compressedBytesForRowRanges( - ParquetFileReader reader, + private long compressedBytesForRowRanges( int blockIndex, Map chunks, List columns, @@ -1224,9 +1254,10 @@ private static long compressedBytesForRowRanges( if (columns == null || columns.isEmpty() || rowRangeCount == 0) { return 0L; } - long blockRowCount = reader.getRowGroups().get(blockIndex).getRowCount(); + long blockRowCount = lateMatReader.getRowGroups().get(blockIndex).getRowCount(); boolean wholeBlock = rowRangeCount == blockRowCount; - ColumnIndexStore ciStore = wholeBlock ? null : reader.getColumnIndexStore(blockIndex); + ColumnIndexStore ciStore = + wholeBlock ? null : lateMatReader.getColumnIndexStore(blockIndex); long total = 0L; for (ColumnDescriptor column : columns) { ColumnPath path = ColumnPath.get(column.getPath()); @@ -1246,6 +1277,19 @@ private static long compressedBytesForRowRanges( continue; } if (offsetIndex == null) { + // The store answers null, rather than throwing, for a path it was not built with. That can + // only mean the block's store was built while a narrower schema was requested than this + // walk asks about, an ordering bug rather than a property of the file, and the footer tells + // the two apart. It is reported rather than thrown: this walk only produces a counter, and + // `ignoreCorruptFiles` turns any exception from a reader into a silently truncated file, so + // a byte metric must not be able to change the answer. + if (chunk.getOffsetIndexReference() != null && !loggedMissingStoreEntry) { + loggedMissingStoreEntry = true; + LOG.warn("Undercounting the storage filter's avoided bytes for {}: column " + + path.toDotString() + " of row group " + blockIndex + " has an offset index the " + + "block's column index store was not built with", + MDC.of(LogKeys.PATH, lateMatReader.getFile())); + } continue; } // The dictionary page is read whenever any data page of the chunk is, so count it here the @@ -1298,16 +1342,19 @@ private RowRanges evaluateStorageFilter( keyDescriptors[i], keyRequired[i], keyPages, convertTz, datetimeRebaseMode, datetimeRebaseTz, int96RebaseMode, int96RebaseTz, writerVersion); } - ensureCurrentKeyAccumulatorsAllocated(); + if (spliceCurrentRowGroup) ensureCurrentKeyAccumulatorsAllocated(); PrimitiveIterator.OfLong rowIndexIter = pushedFilterRanges.iterator(); RowRanges.Builder finalRangesBuilder = RowRanges.builder(); - survivorRangeCount = 0L; + // What this row group retains, weighed against the budget below: the bytes buffered for + // splicing, and the ranges the surviving rows fall into. + long splicedBytes = 0L; + long survivorRangeCount = 0L; long previousSurvivor = -2L; // Recomputed rather than taken from the caller: a count that disagreed with this iterator would // silently drop surviving rows, and no post-scan Filter is left to catch that. long remaining = pushedFilterRanges.rowCount(); - boolean accumulate = true; + long cap = storageFilter.maxSplicedRowGroupBytes(); while (remaining > 0) { int num = (int) Math.min((long) capacity, remaining); for (int i = 0; i < keyScratchVectors.length; i++) { @@ -1317,21 +1364,48 @@ private RowRanges evaluateStorageFilter( keyScratchBatch.setNumRows(num); for (int r = 0; r < num; r++) { long blockRow = rowIndexIter.nextLong(); - if (storageFilter.test(keyScratchBatch.getRow(r))) { + boolean survives; + try { + survives = storageFilter.test(keyScratchBatch.getRow(r)); + } catch (RuntimeException e) { + if (!storageFilter.isEvaluationError(e)) throw e; + // Fail open. The predicate ran on a row that, in the plan, an earlier conjunct would + // have rejected before it, so a plain scan never evaluates it there. Giving the filter up + // for this row group puts every row the pushed filter kept back in the output, and the + // post-scan Filter then evaluates the conjuncts in their own order. Either an earlier one + // drops the row before this expression runs, or it does not and the query fails the way + // it would have without this feature. + logFilterGivenUpOnError(e); + giveUpFilter(); + return null; + } + if (survives) { finalRangesBuilder.addSelectedRow(blockRow); if (blockRow != previousSurvivor + 1) survivorRangeCount++; previousSurvivor = blockRow; - if (accumulate) { - accumulate = appendSurvivorRowToAccumulators(r); - // The ranges being built are about to be thrown away, so stop evaluating the rest. - if (filterGivenUp) return null; + if (spliceCurrentRowGroup) splicedBytes += appendSurvivorRowToAccumulators(r); + // Both halves of what this row group retains grow per survivor, and either can cross + // the budget on its own, so they are weighed together here and nowhere else. The cheaper + // concession comes first: release the buffer, and give the filter up as well if the + // ranges alone still do not fit. + long rangeBytes = rowRangeStateBytes(survivorRangeCount); + if (splicedBytes + rangeBytes > cap) { + if (spliceCurrentRowGroup) { + abandonSplicing(); + splicedBytes = 0L; + } + if (rangeBytes > cap) { + // The ranges being built are about to be thrown away, so stop evaluating the rest. + giveUpFilter(); + return null; + } } } } remaining -= num; } - if (accumulate) { + if (spliceCurrentRowGroup) { finalizePartialAccumulators(); } @@ -1375,13 +1449,9 @@ private void ensureCurrentKeyAccumulatorsAllocated() { /** * Appends row {@code srcRow} of every key column to the accumulators, pushing them onto their * queues once full. All key columns advance in lockstep, which is what keeps the queues aligned. - * - *

Returns false when the buffered survivors have passed - * {@code spark.sql.parquet.storageFilterPushdown.maxSplicedRowGroupBytes} and this row group has - * given splicing up, in which case it has already released what it held and the caller must stop - * calling this. + * Returns the bytes the row added, which the caller weighs against its budget. */ - private boolean appendSurvivorRowToAccumulators(int srcRow) { + private long appendSurvivorRowToAccumulators(int srcRow) { final int dstRow = currentKeyAccumulatorRowCount; final WritableColumnVector[] accs = currentKeyAccumulators; final WritableColumnVector[] srcs = keyScratchVectors; @@ -1399,32 +1469,42 @@ private boolean appendSurvivorRowToAccumulators(int srcRow) { if (keyVariableLength[i]) valueBytes += dst.getArrayLength(dstRow); } } - splicedBytes += keyFixedBytesPerRow + valueBytes; currentKeyAccumulatorRowCount = dstRow + 1; if (currentKeyAccumulatorRowCount == capacity) { - for (int i = 0; i < currentKeyAccumulators.length; i++) { - keyVectorQueues[i].addLast(currentKeyAccumulators[i]); - currentKeyAccumulators[i] = null; - } - // Checked only here, so the cost is one comparison per capacity-sized vector rather than per - // row. A row group whose survivors fit in a single accumulator is never checked at all: it - // then holds one capacity-sized vector per key column, which is what a plain read holds. - // - // Two allocations share the budget, and giving splicing up releases only the first, so the - // cheaper concession comes first: drop the buffer, and give the filter up as well if the row - // ranges alone still do not fit. The second measurement is taken after that concession, so it - // counts the leaves phase 2 will now drive, which is every projected column rather than the - // non-key ones. Either way this row group stops accumulating. - long cap = storageFilter.maxSplicedRowGroupBytes(); - if (splicedBytes + rowRangeStateBytes(survivorRangeCount) > cap) { - abandonSplicing(); - // Splicing is already gone, so the flag is all that is left to set. - if (rowRangeStateBytes(survivorRangeCount) > cap) filterGivenUp = true; - return false; - } + pushAccumulatorsToQueues(); ensureCurrentKeyAccumulatorsAllocated(); } - return true; + return keyFixedBytesPerRow + valueBytes; + } + + /** Hands every accumulator to its queue, which is what the emit path dequeues from. */ + private void pushAccumulatorsToQueues() { + for (int i = 0; i < currentKeyAccumulators.length; i++) { + keyVectorQueues[i].addLast(currentKeyAccumulators[i]); + currentKeyAccumulators[i] = null; + } + } + + /** Releases the pages phase 2 read for the row group just emitted, if any. */ + private void closeDataPages() { + if (dataPages != null) { + dataPages.close(); + dataPages = null; + } + } + + /** + * Reports the first row group of this file whose filter could not be evaluated. Once per file, + * because a file whose values do that tends to do it again, and the row groups that follow are + * still filtered normally. + */ + private void logFilterGivenUpOnError(RuntimeException e) { + if (loggedFilterEvaluationError) return; + loggedFilterEvaluationError = true; + LOG.warn("Reading a row group of {} without the storage filter: evaluating it on a row raised " + + "an error. The filter is still applied above the scan, so the answer is unchanged, and " + + "the remaining row groups are filtered as usual", e, + MDC.of(LogKeys.PATH, lateMatReader.getFile())); } /** @@ -1444,6 +1524,8 @@ private void giveUpFilter() { */ private void abandonSplicing() { for (java.util.ArrayDeque q : keyVectorQueues) { + // A queue can be null if an allocation failed part way through `initializeSplicingState`. + if (q == null) continue; for (WritableColumnVector v : q) v.close(); q.clear(); } @@ -1459,10 +1541,7 @@ private void abandonSplicing() { */ private void finalizePartialAccumulators() { if (currentKeyAccumulatorRowCount == 0) return; - for (int i = 0; i < currentKeyAccumulators.length; i++) { - keyVectorQueues[i].addLast(currentKeyAccumulators[i]); - currentKeyAccumulators[i] = null; - } + pushAccumulatorsToQueues(); currentKeyAccumulatorRowCount = 0; } @@ -1472,16 +1551,13 @@ private void finalizePartialAccumulators() { */ private void closeSplicingState() { keyVectorsPublished = false; + // Releasing everything the splicing path holds is what `abandonSplicing` does, and both arrays + // are set together by `initializeSplicingState`, so one null check covers the state. if (keyVectorQueues != null) { - for (java.util.ArrayDeque q : keyVectorQueues) { - if (q != null) { - for (WritableColumnVector v : q) v.close(); - q.clear(); - } - } + abandonSplicing(); + currentKeyAccumulators = null; + keyVectorQueues = null; } - closeAll(currentKeyAccumulators); - currentKeyAccumulators = null; } /** Closes every non-null vector of {@code vectors}; tolerates a null array. */ diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala index aa5fead84b538..06f5c08342f3e 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala @@ -797,12 +797,26 @@ case class FileSourceScanExec( lazy val inputRDD: RDD[InternalRow] = { val options = relation.options + (FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString) - // Only route through the storage-filter entry point when there is something to push, so that a + // The storage-filter entry point is only asked when there is something to push, so a // `FileFormat` subclass which customizes reading by overriding `buildReaderWithPartitionValues` - // keeps being used on every other query: `ParquetFileFormat` answers - // `buildReaderWithStorageFilters` with a full reader the subclass knows nothing about. + // keeps being used on every other query. A format that declines, which is the default, falls + // back to that builder here rather than inside itself. + val storageFilterReader = if (preparedStorageFilters.isEmpty) { + None + } else { + relation.fileFormat.buildReaderWithStorageFilters( + sparkSession = relation.sparkSession, + dataSchema = relation.dataSchema, + partitionSchema = relation.partitionSchema, + requiredSchema = requiredSchema, + filters = pushedDownFilters, + storageFilters = preparedStorageFilters, + options = options, + hadoopConf = getHadoopConf(relation.sparkSession, relation.options), + storageFilterMetrics = storageFilterMetrics) + } val readFile: (PartitionedFile) => Iterator[InternalRow] = - if (preparedStorageFilters.isEmpty) { + storageFilterReader.getOrElse { relation.fileFormat.buildReaderWithPartitionValues( sparkSession = relation.sparkSession, dataSchema = relation.dataSchema, @@ -811,17 +825,6 @@ case class FileSourceScanExec( filters = pushedDownFilters, options = options, hadoopConf = getHadoopConf(relation.sparkSession, relation.options)) - } else { - relation.fileFormat.buildReaderWithStorageFilters( - sparkSession = relation.sparkSession, - dataSchema = relation.dataSchema, - partitionSchema = relation.partitionSchema, - requiredSchema = requiredSchema, - filters = pushedDownFilters, - storageFilters = preparedStorageFilters, - options = options, - hadoopConf = getHadoopConf(relation.sparkSession, relation.options), - storageFilterMetrics = storageFilterMetrics) } val readRDD = if (bucketedScan) { diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala index 3f58fb6c7cad6..c7c206e727d22 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala @@ -173,15 +173,17 @@ trait FileFormat { * * Honoring them is optional, here and in a reader that does implement them: the planner leaves * every one of them in the post-scan `Filter` as well, so ignoring one is a missed optimization - * rather than a wrong answer. That is why this default can simply delegate to - * [[buildReaderWithPartitionValues]]. + * rather than a wrong answer. * - * A format that supports storage-filter pushdown overrides this, and must not let the two - * builders call each other. This default delegates one way, and it delegates on `this`, so an - * override of [[buildReaderWithPartitionValues]] that delegates back here closes the loop and - * recurses until the driver's stack runs out. Calling this default through `super` is part of - * that loop, not an escape from it. Route both to a private implementation instead, the way - * `ParquetFileFormat` does. + * Being optional is also an obligation. A storage filter is evaluated out of the plan's order, + * without the conjuncts that precede it, so it can raise an error on a row those conjuncts would + * have rejected, which is an error a plain scan never raises. A reader must not fail the query + * for that: it gives the filter up for as much of the read as it needs to and lets the post-scan + * `Filter` decide, in its own order. + * + * A format that does not apply storage filters returns `None`, which is the default, and the + * caller then builds an ordinary reader. Returning an `Option` rather than delegating from here + * is what keeps the two builders from being able to call each other. * * Scalar subqueries inside `storageFilters` are expected to have been materialized before this * method is called, so that the returned reader can be safely serialized to executors. @@ -200,10 +202,14 @@ trait FileFormat { options: Map[String, String], hadoopConf: Configuration, storageFilterMetrics: Map[String, SQLMetric] = Map.empty - ): PartitionedFile => Iterator[InternalRow] = { - buildReaderWithPartitionValues( - sparkSession, dataSchema, partitionSchema, requiredSchema, filters, options, hadoopConf) - } + ): Option[PartitionedFile => Iterator[InternalRow]] = None + + /** + * Whether this format applies storage filters in this session at all, which is also where the + * conf that enables them belongs: a format's own conf should not decide for another format. Asked + * once per scan, before anything per conjunct, so a format that answers false costs one call. + */ + def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean = false /** * Whether this format's reader can evaluate `expr` as a storage filter, i.e. whether the planner diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala index 300f0d80a10d8..bc15827c856b0 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala @@ -162,7 +162,9 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { * to hold their row ranges, without the answer depending on it. * * Two of the conditions are per scan, and failing either offers nothing: - * - The storage-filter pushdown SQL conf is on. + * - [[FileFormat.supportsStorageFilterPushdown]] holds. That is where a format reads the conf + * that enables this, so a format's own conf never decides for another format, and asking it + * first keeps everything below off the path of a scan that will not use it. * - [[FileFormat.supportBatch]] holds for the schema the reader will see, * `partitionSchema ++ outputDataSchema`, which is the schema a format's reader builder derives * its own vectorized-read decision from. Late materialization needs a batch read, so this asks @@ -189,17 +191,19 @@ object FileSourceStrategy extends Strategy with PredicateHelper with Logging { readDataColumns: Seq[Attribute], outputDataSchema: StructType): Seq[Expression] = { val sparkSession = fsRelation.sparkSession - val sqlConf = sparkSession.sessionState.conf - if (!sqlConf.parquetStorageFilterPushdownEnabled) return Nil + if (!fsRelation.fileFormat.supportsStorageFilterPushdown(sparkSession)) return Nil val resultSchema = StructType(fsRelation.partitionSchema.fields ++ outputDataSchema.fields) if (!fsRelation.fileFormat.supportBatch(sparkSession, resultSchema)) return Nil val dataAttrs = AttributeSet(readDataColumns) - val offered = afterScanFilters.filter { expr => + // Over `toSeq` rather than the set: `ExpressionSet.filter` hands the predicate + // `e.canonicalized`, which drops attribute names and metadata, and a format deciding by either + // would answer about an expression it will never be given. + val offered = afterScanFilters.toSeq.filter { expr => val refs = expr.references expr.deterministic && refs.nonEmpty && refs.forall(dataAttrs.contains) && fsRelation.fileFormat.supportsStorageFilter(expr) - }.toSeq + } val keyAttrs = AttributeSet(offered.flatMap(_.references)) if (readDataColumns.forall(keyAttrs.contains)) Nil else offered } diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala index eaa47ac395302..9fedf7ca03d67 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala @@ -198,13 +198,20 @@ class ParquetFileFormat } /** + * The conf that turns this on is read here rather than in the planner, the way `supportBatch` + * reads its own confs, so a Parquet-named conf does not decide for a format that is not Parquet. + * * Subclasses answer false on purpose, even though they inherit this reader: a subclass may * customize reading by overriding `buildReaderWithPartitionValues`, and a scan with storage * filters routes through `buildReaderWithStorageFilters` instead, which would silently bypass * whatever the subclass does. */ + override def supportsStorageFilterPushdown(sparkSession: SparkSession): Boolean = + getSqlConf(sparkSession).parquetStorageFilterPushdownEnabled && + getClass == classOf[ParquetFileFormat] + override def supportsStorageFilter(expr: Expression): Boolean = - getClass == classOf[ParquetFileFormat] && ParquetStorageFilter.isSupportedStorageFilter(expr) + ParquetStorageFilter.isSupportedStorageFilter(expr) override def buildReaderWithStorageFilters( sparkSession: SparkSession, @@ -215,15 +222,13 @@ class ParquetFileFormat storageFilters: Seq[Expression], options: Map[String, String], hadoopConf: Configuration, - storageFilterMetrics: Map[String, SQLMetric]): PartitionedFile => Iterator[InternalRow] = { - buildParquetReader(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, - storageFilters, options, hadoopConf, storageFilterMetrics) + storageFilterMetrics: Map[String, SQLMetric]) + : Option[PartitionedFile => Iterator[InternalRow]] = { + Some(buildParquetReader(sparkSession, dataSchema, partitionSchema, requiredSchema, filters, + storageFilters, options, hadoopConf, storageFilterMetrics)) } - /** - * The implementation behind both public entry points above, which is why neither of them calls - * the other. See the warning on `FileFormat.buildReaderWithStorageFilters`. - */ + /** The implementation behind both public entry points above. */ private def buildParquetReader( sparkSession: SparkSession, dataSchema: StructType, @@ -296,7 +301,7 @@ class ParquetFileFormat FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, null), bytesAvoidedByPageFiltering = storageFilterMetrics.getOrElse( FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING, null)) - // `create` requires every condition extractStorageFilters already pre-checked, so a violation + // `create` requires every condition storageFiltersFor already pre-checked, so a violation // is a planner bug rather than something to work around here. Some(ParquetStorageFilter.create(storageFilters, requiredSchema, metrics, sqlConf.parquetStorageFilterPushdownMaxSplicedRowGroupBytes)) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala index e7d27fa863842..a5425ba30966a 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilter.scala @@ -19,8 +19,10 @@ package org.apache.spark.sql.execution.datasources.parquet import java.util.concurrent.ConcurrentHashMap +import org.apache.spark.SparkThrowable import org.apache.spark.sql.catalyst.InternalRow -import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, ExprUtils, Literal, Predicate, XxHash64} +import org.apache.spark.sql.catalyst.expressions.{And, BasePredicate, BloomFilterMightContain, BoundReference, Expression, Literal, Predicate, XxHash64} +import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION import org.apache.spark.sql.execution.metric.SQLMetric import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, DayTimeIntervalType, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, StructType, TimestampNTZType, TimestampType, TimeType, YearMonthIntervalType} @@ -133,9 +135,28 @@ class ParquetStorageFilter private ( * `true` iff the predicate is literally true; a null or false result is interpreted as "drop * every row" by the reader. */ - def evalAllMissing(): Boolean = { + def evalAllMissing(): Option[Boolean] = { require(keyColumnIndices.isEmpty, "evalAllMissing only valid when all key columns are missing") - boundExpression.eval(InternalRow.empty) == true + // The substituted constant can be one this expression throws on, the same way a row's value can + // be, so the same rule applies: fail open. None means the reader must not decide from this + // filter at all and has to read the file the way a plain scan would. + try { + Some(boundExpression.eval(InternalRow.empty) == true) + } catch { + case e: RuntimeException if isEvaluationError(e) => None + } + } + + /** + * Whether `e` is an error from evaluating the predicate rather than a defect in the reader. Only + * the first kind may be swallowed, and the error class is what tells them apart: a value produces + * one (an invalid cast under ANSI, an overflow, a division by zero), while a null dereference or + * a failed assertion produces none, and an internal error says so in the class itself. Asked of + * an instance rather than of the companion, so the Java reader can call it plainly. + */ + private[parquet] def isEvaluationError(e: RuntimeException): Boolean = e match { + case t: SparkThrowable => !t.isInternalError + case _ => false } } @@ -147,7 +168,7 @@ object ParquetStorageFilter { * `[0, requestedSchema.length)`). Multiple filters are combined with logical AND, so a row must * satisfy all of them to survive. * - * Every condition below is asserted rather than handled: `extractStorageFilters` pre-checks all + * Every condition below is asserted rather than handled: `storageFiltersFor` pre-checks all * of them, so a violation here is a planner bug. A reader giving a filter up at read time is a * different matter. These conditions are about the filter being well formed at all. * @@ -231,29 +252,28 @@ object ParquetStorageFilter { // NOT: the reader evaluates the expression it is given and treats a false as "drop this row". // Every reference is checked, not just the ones on the value side, because [[create]] binds // and type-checks all of them. - bloom.references.forall(a => isSupportedKeyType(a.dataType)) && canEvaluateOnEveryRow(bloom) + bloom.references.forall(a => isSupportedKeyType(a.dataType)) && canEvaluateInTheReader(bloom) case _ => false } /** - * Whether the reader may evaluate `bloom`'s value side on any row of the files it reads. - * - * It has to ask, because the reader evaluates the predicate on every row the pushed data filter - * left, while in the plan the conjunct ran after the ones ahead of it and was skipped for the - * rows they rejected. A `CAST(s AS BIGINT)` key, which `InjectRuntimeFilter` builds for a - * string-to-long join, then throws in ANSI mode on a row an earlier conjunct would have dropped, - * a query that succeeds without this feature. + * Whether the reader can evaluate this bloom's value side at all. * - * So the value side must be a hash of expressions that cannot fail on any input, which - * [[ExprUtils.canEvaluateUnconditionally]] decides. `XxHash64` and the membership test itself are - * total for every type they accept at analysis time. + * It does not have to be an expression that is safe to evaluate on every row. The reader + * evaluates the predicate without the conjuncts that precede it in the plan, so an expression + * that throws on a row an earlier conjunct would have rejected throws where a plain scan does + * not. That is + * handled where it arises rather than here: the reader gives the filter up for the row group and + * reads it plainly, and the post-scan `Filter` then evaluates every conjunct in its own order. * - * This is a question about the conjunct as the planner holds it, with attribute references for - * leaves. It is not one to ask of a bound expression, which that whitelist does not admit. + * What is left is what the reader cannot evaluate at all. A subquery has no plan to run on an + * executor, and a non-deterministic expression needs the `initialize(partitionIndex)` that + * `GeneratePredicate` emits for it and the reader never calls. */ - private def canEvaluateOnEveryRow(bloom: BloomFilterMightContain): Boolean = + private def canEvaluateInTheReader(bloom: BloomFilterMightContain): Boolean = bloom.valueExpression match { - case hash: XxHash64 => hash.children.forall(ExprUtils.canEvaluateUnconditionally) + case hash: XxHash64 => + hash.children.forall(c => c.deterministic && !c.containsPattern(PLAN_EXPRESSION)) case _ => false } } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala index 65aa8d8b9240a..943104fd2095f 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala @@ -28,8 +28,14 @@ import scala.jdk.CollectionConverters._ import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, FSInputStream, Path, RawLocalFileSystem} import org.apache.hadoop.mapreduce.Job -import org.apache.parquet.column.Encoding -import org.apache.parquet.hadoop.{ParquetFileReader, ParquetInputFormat, ParquetOutputFormat} +import org.apache.parquet.column.{Encoding, ParquetProperties} +import org.apache.parquet.column.impl.ColumnWriteStoreV1 +import org.apache.parquet.column.page.DataPageV1 +import org.apache.parquet.column.page.mem.MemPageStore +import org.apache.parquet.hadoop.{ParquetFileReader, ParquetFileWriter, ParquetInputFormat, ParquetOutputFormat} +import org.apache.parquet.hadoop.metadata.{ColumnChunkMetaData, CompressionCodecName} +import org.apache.parquet.hadoop.util.HadoopOutputFile +import org.apache.parquet.schema.MessageTypeParser import org.apache.spark.paths.SparkPath import org.apache.spark.sql.{sources, DataFrame, QueryTest, Row, SparkSession} @@ -37,6 +43,7 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.{And, Attribute, AttributeReference, BloomFilterMightContain, BoundReference, Cast, Coalesce, EqualTo, Expression, GreaterThanOrEqual, IsNull, LessThanOrEqual, Literal, Or, Predicate, Rand, Remainder, XxHash64} import org.apache.spark.sql.catalyst.plans.logical.{Filter => LogicalFilter} import org.apache.spark.sql.execution.{CollapseCodegenStages, ColumnarToRowExec, FileSourceScanExec, FileSourceScanLike, FilterExec, LocalLimitExec, SparkPlan, WholeStageCodegenExec} +import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import org.apache.spark.sql.execution.datasources.{FileFormat, FileSourceStrategy, OutputWriterFactory, PartitionedFile} import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics} import org.apache.spark.sql.functions.col @@ -44,6 +51,7 @@ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.SharedSparkSession import org.apache.spark.sql.types._ import org.apache.spark.sql.vectorized.{ColumnarBatch, ColumnVector} +import org.apache.spark.unsafe.types.UTF8String import org.apache.spark.util.Utils import org.apache.spark.util.sketch.BloomFilter @@ -52,7 +60,8 @@ import org.apache.spark.util.sketch.BloomFilter * [[ParquetStorageFilter]]. Writes small multi-row-group parquet files, wires a hand-built filter * into the reader, and asserts correctness + the two storage-filter metrics. */ -class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { +class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession + with AdaptiveSparkPlanHelper { import testImplicits._ // Writes `df` as one parquet file under a fresh directory and returns its path. Every write @@ -282,8 +291,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { test("multi-batch emit: survivor count exceeds capacity") { // Drive the reader at capacity = 16 with a row group of 100 surviving rows. Exercises: // - The per-key-column queue holding multiple full-capacity vectors plus a partial tail. - // - pendingCloseKeyVectors getting closed at the start of every subsequent emit. - // - The per-emit ColumnarBatch reconstruction running ceil(100/16) = 7 times. + // - The published queue head getting closed at the start of every subsequent emit. + // - The batch's key slots being rewritten ceil(100/16) = 7 times. withTempDir { dir => val keys = (1L to 100L) // Big rowGroupSize so all 100 rows fit in one row group. @@ -336,7 +345,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } test("ParquetStorageFilter.create rejects a filter that violates a planner precondition") { - // These are all planner bugs by construction: extractStorageFilters pre-checks each one, and + // These are all planner bugs by construction: storageFiltersFor pre-checks each one, and // by the time create runs the conjunct is gone from the post-scan Filter, so a soft rejection // would silently return rows the filter excludes. create fails instead. val requested = StructType(Seq( @@ -406,12 +415,12 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val dropped = rewriteWithBloomContaining(xxHash64(42L, LongType)) assert(dropped.keyColumnIndices.isEmpty, "all key positions are missing; keyColumnIndices should be empty") - assert(!dropped.evalAllMissing(), + assert(dropped.evalAllMissing().contains(false), "the null key's hash is not in the bloom, so evalAllMissing must return false " + "(skip the file)") val kept = rewriteWithBloomContaining(nullKeyHash) - assert(kept.evalAllMissing(), + assert(kept.evalAllMissing().contains(true), "the null key's hash IS in the bloom, so evalAllMissing must return true (keep the file)") } @@ -428,13 +437,13 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { new XxHash64(Seq(BoundReference(0, LongType, nullable = true)))) // Substituting the default keeps the file, because the default's hash is in the bloom. assert(ParquetStorageFilter.create(Seq(expr), requested) - .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing(), + .rewriteForMissingKeys(Array(0), Array(defaultValue)).evalAllMissing().contains(true), "substituting the existence default must probe the default's hash and keep the file") // Substituting null instead would drop it, the bug this guards against. A second filter, // because a rewrite is cached per set of missing positions: one scan always substitutes the // same values for them. - assert(!ParquetStorageFilter.create(Seq(expr), requested) - .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing(), + assert(ParquetStorageFilter.create(Seq(expr), requested) + .rewriteForMissingKeys(Array(0), Array(null)).evalAllMissing().contains(false), "sanity check: substituting null probes a different hash and would drop the file") } @@ -460,10 +469,35 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val rewritten = filter.rewriteForMissingKeys(Array(0), Array(null)) assert(rewritten.keyColumnIndices.isEmpty, "all key positions are missing; keyColumnIndices should be empty") - assert(rewritten.evalAllMissing(), + assert(rewritten.evalAllMissing().contains(true), "Coalesce(null, default) yields default; bloom hit, so evalAllMissing must return true") } + test("evalAllMissing fails open, and a reader defect is not swallowed") { + // The constant substituted for a missing key column can be one the predicate throws on, the + // same way a row's value can be, so the same rule applies: fail open and let the post-scan + // Filter decide. An empty answer is what tells the reader to read the file plainly instead of + // skipping it, which is the difference between a missed saving and a lost file. + withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") { + val requested = StructType(Seq(StructField("k", StringType, nullable = true))) + val castKey = GreaterThanOrEqual( + Cast(BoundReference(0, StringType, nullable = true), LongType), Literal(1L)) + val filter = ParquetStorageFilter.create(Seq(castKey), requested) + .rewriteForMissingKeys(Array(0), Array(UTF8String.fromString("not-a-number"))) + assert(filter.evalAllMissing().isEmpty, + "an error while evaluating the substituted constant must not decide about the file") + + // And the judgement that separates a value error from a defect, in both directions. + val castError = intercept[RuntimeException] { + Cast(Literal.create("not-a-number"), LongType).eval(InternalRow.empty) + } + assert(filter.isEvaluationError(castError), + s"an invalid cast is a value error; got ${castError.getClass.getName}") + assert(!filter.isEvaluationError(new IllegalStateException("a reader defect")), + "a defect carries no error class, so it has to fail the query rather than the filter") + } + } + test("rewriteForMissingKeys: partial-missing keys narrow keyColumnIndices and renumber") { // Two keys, one missing, one present. Verify the present key's BoundReference is renumbered to // position 0 in the new layout, and the missing one is substituted with Literal(null). We don't @@ -576,8 +610,10 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // ----- FileSourceStrategy bloom-filter extraction ----- // Counts BloomFilterMightContain expressions inside FilterExec nodes of a physical plan. + // `collect` from AdaptiveSparkPlanHelper rather than SparkPlan's, which stops at an + // `AdaptiveSparkPlanExec` and would report zero for every plan AQE wrapped. private def countBloomFiltersInPostScanFilters(plan: SparkPlan): Int = { - plan.collect { + collect(plan) { case f: FilterExec => f.condition.collect { case _: BloomFilterMightContain => 1 }.sum }.sum @@ -585,7 +621,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // Counts BloomFilterMightContain expressions inside FileSourceScanExec.storageFilters. private def countBloomFiltersInStorageFilters(plan: SparkPlan): Int = { - plan.collect { + collect(plan) { case s: FileSourceScanExec => s.storageFilters.map(_.collect { case _: BloomFilterMightContain => 1 }.sum).sum }.sum @@ -739,9 +775,9 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // second time, so it transfers strictly more bytes. Without that second assertion the test // would pass even if the cap never reached the reader. // - // The batch size is what makes the cap reachable at all: the reader examines the count only - // once a batch worth of survivors per key column has been buffered, so 51 survivors have to - // cross that line more than once. + // The batch size is not what makes the cap reachable, since the count is examined after + // every surviving row. It is here because a row group read the plain way past the cap still + // has to emit in batches, and 51 survivors over a capacity of 16 span several of them. // // The `k` projection is the interesting one: an all-keys projection normally skips phase 2 // entirely, so past the cap it has to read the key column there like any other column. @@ -767,6 +803,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters( spark, fileSchema, new StructType(), readSchema, Nil, storageFilters, Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, Map.empty) + .getOrElse(fail("ParquetFileFormat must answer with a reader")) val file = PartitionedFile( InternalRow.empty, SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"), @@ -799,8 +836,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { // The emitted batch is one object for the whole read, so its key slots have to follow the path // each row group took. This file makes all four cases occur in order, at 100 rows per row group // and a batch capacity of 16: - // - rows 1-100: 8 survivors, fewer than the capacity, so the cap is never weighed and the row - // group splices; + // - rows 1-100: 8 survivors, which buffer 72 bytes and fall in one range of 40, inside the + // 150-byte cap, so the row group splices; // - rows 101-200: 51 survivors, so the buffer passes the cap and phase 2 reads every projected // column. The key slots must go back to the persistent vectors here, or the batch reads keys // out of a vector the previous row group's last batch already released; @@ -823,7 +860,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true))), StorageFilterMetrics(rowGroupsSkipped = rgSkipped), - maxSplicedRowGroupBytes = 100L) + maxSplicedRowGroupBytes = 150L) val (result, reader) = readAllWith(path, Seq("k", "v"), filter, (batch, i) => (batch.column(0).getLong(i), batch.column(1).getUTF8String(i).toString), capacity = 16) @@ -853,29 +890,137 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { StructField("k", LongType, nullable = true), StructField("v", StringType, nullable = true))) - def avoidedBytes(cap: Long): Long = { + def avoidedBytes(cap: Long, threshold: Long): Long = { val pf = SQLMetrics.createSizeMetric(spark.sparkContext, "bytesAvoidedByPf") val filter = ParquetStorageFilter.create( - Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(350L))), + Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = true), Literal(threshold))), fileSchema, StorageFilterMetrics(bytesAvoidedByPageFiltering = pf), cap) val (_, reader) = readAllWith(path, Seq("k", "v"), filter, (b, i) => b.column(0).getLong(i), capacity = 16) try pf.value finally reader.close() } - val spliced = avoidedBytes(64L * 1024 * 1024) - val plain = avoidedBytes(100L) + val spliced = avoidedBytes(64L * 1024 * 1024, 350L) + val plain = avoidedBytes(100L, 350L) assert(spliced > 0, s"page filtering has to avoid something here; got $spliced") assert(plain < spliced, s"the second key read must count against the saving; plain=$plain spliced=$spliced") + + // The two halves of the budget are weighed together, and this is the case that says so: eight + // survivors of a long key buffer 8 * (1 + 8) = 72 bytes and fall in one range of 40, so + // neither half reaches the 100-byte cap on its own while the sum passes it at the seventh. + // Weighing them separately would keep splicing here, and report the larger saving for it. + val eightSpliced = avoidedBytes(64L * 1024 * 1024, 393L) + val eightPlain = avoidedBytes(100L, 393L) + assert(eightSpliced > 0, s"and avoid something with eight survivors; got $eightSpliced") + assert(eightPlain < eightSpliced, + s"the sum of the two halves must cross the cap; plain=$eightPlain spliced=$eightSpliced") + } + } + + // Writes a parquet file the low-level way, with no offset index for any column. The + // `ParquetFileWriter.writeDataPage` overloads that take no row count use parquet's no-op offset + // index builder, which is how a writer other than parquet-mr's own produces a file whose row + // groups cannot be read in part. Every column is a required int64, so one page writer serves all. + private def writeParquetFileWithoutOffsetIndex( + dir: File, + blocks: Seq[Seq[(Long, Long)]]): String = { + val schema = MessageTypeParser.parseMessageType( + "message spark_schema { required int64 k; required int64 v; }") + val file = new File(dir, s"no-offset-index-${System.nanoTime()}.parquet") + val hadoopPath = new Path(file.getAbsolutePath) + val writer = new ParquetFileWriter( + HadoopOutputFile.fromPath(hadoopPath, spark.sessionState.newHadoopConf()), + schema, ParquetFileWriter.Mode.CREATE, 128L * 1024 * 1024, 8) + writer.start() + blocks.foreach { block => + writer.startBlock(block.size) + schema.getColumns.asScala.zipWithIndex.foreach { case (cd, colIdx) => + val pageStore = new MemPageStore(block.size) + val writeStore = new ColumnWriteStoreV1(pageStore, + ParquetProperties.builder().withPageSize(256).withDictionaryEncoding(false).build()) + val columnWriter = writeStore.getColumnWriter(cd) + block.foreach { row => + columnWriter.write(if (colIdx == 0) row._1 else row._2, 0, 0) + writeStore.endRecord() + } + writeStore.flush() + writer.startColumn(cd, block.size, CompressionCodecName.UNCOMPRESSED) + val pageReader = pageStore.getPageReader(cd) + var written = 0L + while (written < block.size) { + val page = pageReader.readPage().asInstanceOf[DataPageV1] + writer.writeDataPage(page.getValueCount, page.getUncompressedSize, page.getBytes, + page.getStatistics, page.getRlEncoding, page.getDlEncoding, page.getValueEncoding) + written += page.getValueCount + } + writer.endColumn() + } + writer.endBlock() } + writer.end(new java.util.HashMap[String, String]()) + file.getAbsolutePath } - test("a row group whose survivors scatter into too many ranges is read without the filter") { - // `finalRanges` is built one row at a time, so an alternating filter makes one range per - // surviving row, and every column reader phase 2 drives materializes that list for the row - // group. Past the cap the filter is given up for that row group and every one of its rows is - // emitted, which the post-scan Filter then narrows. + test("a file written with no offset index is read with the filter given up") { + // Reading part of a row group needs an offset index, so a file without one cannot be filtered + // at page level. What it can still do is skip a row group the filter empties, which needs no + // index at all, and that is the split this test pins. The three row groups are ordered so that + // the file's first partial read is what discovers the missing index, and the two row groups + // after that discovery still get phase 1: the middle one is emptied by the filter and skipped + // whole, the last keeps rows and is read whole. The missing index is learned from the read that + // fails, so the first row group is the one that discovers it. Spark's own writer always writes + // the page index, hence the hand-built file. + withTempDir { dir => + val blocks = Seq( + (1L to 100L).map(i => (i, i * 10)), + (101L to 200L).map(i => (i, i * 10)), + (201L to 300L).map(i => (i, i * 10))) + val path = writeParquetFileWithoutOffsetIndex(dir, blocks) + // The fixture's whole point, asserted rather than assumed. + assert(footerChunks(path).forall(_.getOffsetIndexReference == null), + "no chunk of the hand-built file may have an offset index") + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val rowsRg = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedByRowGroup") + val rowsPf = SQLMetrics.createMetric(spark.sparkContext, "rowsExcludedWithinRowGroup") + val requested = StructType(Seq( + StructField("k", LongType, nullable = false), + StructField("v", LongType, nullable = false))) + // Keeps part of the first row group, none of the second, part of the third. + val k = BoundReference(0, LongType, nullable = false) + val filter = ParquetStorageFilter.create( + Seq(Or( + And(GreaterThanOrEqual(k, Literal(50L)), LessThanOrEqual(k, Literal(60L))), + And(GreaterThanOrEqual(k, Literal(250L)), LessThanOrEqual(k, Literal(260L))))), + requested, + StorageFilterMetrics( + rowGroupsSkipped = rgSkipped, + rowsExcludedByRowGroup = rowsRg, + rowsExcludedWithinRowGroup = rowsPf)) + val (result, reader) = readAllWith(path, Seq("k", "v"), filter, + (b, i) => (b.column(0).getLong(i), b.column(1).getLong(i))) + try { + assert(result == blocks(0) ++ blocks(2), + s"the emptied row group is skipped and the other two read whole; got ${result.size} rows") + assert(rgSkipped.value == 1 && rowsRg.value == 100, + s"one row group skipped whole, with its rows counted; got ${rgSkipped.value} and " + + s"${rowsRg.value}") + assert(rowsPf.value == 0, + s"and nothing excluded inside a row group, which needs the index; got ${rowsPf.value}") + } finally { + reader.close() + } + } + } + + test("scattered survivors: phase 2 lines its values up with the spliced keys, or gives up") { + // An alternating filter makes one range per surviving row, which is the shape that exercises + // phase 2's range walk and the only one where a defect there is invisible from the row count: + // the keys come from the splice queues and would still be right while the values came from + // other rows. So this asserts the pairs. + // + // Past the cap those ranges cost more than the budget allows, and the filter is given up for + // that row group: every one of its rows is emitted and the post-scan Filter narrows them. withTempDir { dir => val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) @@ -885,34 +1030,73 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val everyOtherRow = EqualTo( Remainder(BoundReference(0, LongType, nullable = true), Literal(2L)), Literal(0L)) - def rowsWithCap(cap: Long, capacity: Int = 4096): Int = { + def readWithCap(cap: Long, capacity: Int = 4096): Seq[(Long, String)] = { val filter = ParquetStorageFilter.create(Seq(everyOtherRow), fileSchema, StorageFilterMetrics(), cap) - val (emitted, reader) = - readAllWith(path, Seq("k", "v"), filter, (b, i) => b.getRow(i).getLong(0), capacity) - try emitted.size finally reader.close() + val (emitted, reader) = readAllWith(path, Seq("k", "v"), filter, + (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString), capacity) + try emitted finally reader.close() } - assert(rowsWithCap(64L * 1024 * 1024) == 200, - "with room for the ranges, only the surviving rows come back") - // The budget is weighed in two places, and the capacity decides which one fires. At 4096 the - // 200 survivors never fill an accumulator, so only the check after phase 1 can catch them. At - // 16 the in-loop check is reached first, and it has to abandon the buffer and give the filter - // up together: keeping one without the other would leave phase 2 reading non-key columns - // while emit spliced keys from a queue holding only the first survivors. - assert(rowsWithCap(1024L) == 400, - "past the cap after phase 1, the filter is given up and the row group comes back whole") - assert(rowsWithCap(1024L, capacity = 16) == 400, - "and the same past the cap inside phase 1's survivor loop") + assert(readWithCap(64L * 1024 * 1024) == rows.filter(_._1 % 2 == 0), + "with room for the ranges, every surviving row comes back with its own value") + // The capacity decides where the budget is crossed. At 4096 the 200 survivors never fill an + // accumulator, so the ranges alone cross it; at 16 the buffer has been through several + // accumulators by then. Both have to abandon the buffer and give the filter up together: + // keeping one without the other would leave phase 2 reading non-key columns while emit + // spliced keys from a queue holding only the first survivors. + assert(readWithCap(1024L) == rows, + "past the cap the filter is given up and the row group comes back whole") + assert(readWithCap(1024L, capacity = 16) == rows, + "and the same when the buffer has been rolled over first") + } + } + + test("a row group that gives the filter up does not take the next one with it") { + // `filterGivenUp` is per row group, reset at the top of each one, so a row group whose ranges + // pass the budget must not disarm the filter for the rest of the file. The first row group here + // scatters into 50 ranges and gives up, the last has its survivors in one run and keeps + // filtering, and the two in between are emptied by the filter and skipped whole. + withTempDir { dir => + val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) + val path = writeParquetFile(dir, rows, rowGroupSize = 256L) + val k = BoundReference(0, LongType, nullable = true) + val scatteredInFirst = And(LessThanOrEqual(k, Literal(100L)), + EqualTo(Remainder(k, Literal(2L)), Literal(0L))) + val runInLast = And(GreaterThanOrEqual(k, Literal(301L)), LessThanOrEqual(k, Literal(320L))) + val rgSkipped = SQLMetrics.createMetric(spark.sparkContext, "rowGroupsSkipped") + val filter = ParquetStorageFilter.create( + Seq(Or(scatteredInFirst, runInLast)), + StructType(Seq( + StructField("k", LongType, nullable = true), + StructField("v", StringType, nullable = true))), + StorageFilterMetrics(rowGroupsSkipped = rgSkipped), + maxSplicedRowGroupBytes = 1024L) + val (result, reader) = readAllWith(path, Seq("k", "v"), filter, + (b, i) => (b.column(0).getLong(i), b.column(1).getUTF8String(i).toString)) + try { + // The first row group comes back whole, the last only its surviving run. + val expected = rows.filter(_._1 <= 100L) ++ rows.filter(r => r._1 >= 301L && r._1 <= 320L) + assert(result == expected, + s"expected the given-up row group whole and the last filtered; got ${result.size} rows") + assert(rgSkipped.value == 2, + s"and the two emptied row groups skipped; got ${rgSkipped.value}") + } finally { + reader.close() + } } } - test("ANSI mode: a bloom over a cast join key stays in the post-scan Filter") { + test("ANSI mode: a cast join key is pushed, and an evaluation error gives the row group up") { // `InjectRuntimeFilter` hashes the join key, so a string-to-long join hands the bloom a // `CAST(s AS BIGINT)`, which throws in ANSI mode on a row whose string is not a number. In the - // plan the bloom runs after the conjunct that excludes such rows, while a reader would evaluate - // it on every row of the ranges the pushed filter left. Extraction therefore has to decline, or - // this query would start failing when the conf is turned on. + // plan the bloom runs after the conjunct that excludes such rows, while the reader evaluates it + // on every row of the ranges the pushed filter left, so it does meet that row. + // + // It is pushed all the same. When the evaluation throws, the reader gives the filter up for + // that row group, and the post-scan Filter evaluates the conjuncts in their own order, where + // `kind = 'num'` drops the row before the cast runs. So the query returns its rows instead of + // failing, which is what it does with the feature off. withTable("ansi_app", "ansi_build") { // One file, one row group, with the non-numeric row inside it: a separate file would be // pruned whole by the pushed `kind = 'num'` filter and the reader would never see the row. @@ -935,8 +1119,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val plan = df.queryExecution.executedPlan assert(countBloomFiltersInPostScanFilters(plan) >= 1, s"a bloom is expected above the scan, otherwise this proves nothing.\nPlan:\n$plan") - assert(countBloomFiltersInStorageFilters(plan) == 0, - s"a cast key must not be extracted.\nPlan:\n$plan") + assert(countBloomFiltersInStorageFilters(plan) == 1, + s"the cast key is pushed.\nPlan:\n$plan") assert(df.collect().map(_.getString(0)).toSet == Set("5"), "and the query must run rather than fail on the non-numeric row") } @@ -976,8 +1160,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { s"postScan=$deterministicPostScan") // `Rand` contributes no reference, so `k` is still the only key column. Two gates reject - // this one now: the planner's `deterministic` test, and `canEvaluateUnconditionally` inside - // the format's answer, whose whitelist is deterministic expressions only. + // this one: the planner's `deterministic` test on the conjunct, and the format's own test + // on the hash's children, since the reader never calls `initialize(partitionIndex)`. val nonDeterministic = new XxHash64(Seq(k, Rand(Literal(1L)))) assert(!nonDeterministic.deterministic, "the value expression must be non-deterministic") val (inScan, postScan) = extractedBlooms(nonDeterministic) @@ -1011,20 +1195,24 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { dictionary = dictionary) } - // The encodings parquet actually used, over every column chunk of the file. Asserted rather than - // assumed, because asking for dictionary encoding does not mean getting it. - private def encodingsOf(filePath: String): Set[Encoding] = { + // Every column chunk of the file, from its footer. The facts the tests below assert about a + // fixture are read off these rather than assumed from the write options. + private def footerChunks(filePath: String): Seq[ColumnChunkMetaData] = { val reader = ParquetFileReader.open(spark.sessionState.newHadoopConf(), new Path(filePath)) try { - reader.getFooter.getBlocks.asScala - .flatMap(_.getColumns.asScala) - .flatMap(_.getEncodings.asScala) - .toSet + reader.getFooter.getBlocks.asScala.flatMap(_.getColumns.asScala).toSeq } finally { reader.close() } } + private def hasOffsetIndexes(filePath: String): Boolean = + footerChunks(filePath).forall(_.getOffsetIndexReference != null) + + // Asserted rather than assumed, because asking for dictionary encoding does not mean getting it. + private def encodingsOf(filePath: String): Set[Encoding] = + footerChunks(filePath).flatMap(_.getEncodings.asScala).toSet + // Reads every batch, projecting each row through `extract`. `storageFilter` may be null, which // selects the plain (non-splicing) vectorized path. Every read helper in this suite goes through // here. @@ -1399,10 +1587,8 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { assert(collected.toSeq == rows.filter(_._1 >= 195L), s"expected exact filtering; got ${collected.map(_._1)}") // A mixed projection has non-key bytes to avoid, in both the skipped row groups and the - // partially kept one. Both counters must be non-negative, and together positive. - assert(bytesAvoidedRg.value >= 0 && bytesAvoidedPf.value >= 0, - s"avoided-byte metrics must never go negative; got rg=${bytesAvoidedRg.value} " + - s"pf=${bytesAvoidedPf.value}") + // partially kept one. Asserting they are non-negative would prove nothing, since + // `SQLMetric.add` drops a negative and the value cannot go below zero. assert(bytesAvoidedRg.value + bytesAvoidedPf.value > 0, "a mixed projection with skipped row groups should avoid some non-key bytes") } finally { @@ -1413,8 +1599,9 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } test("row-at-a-time path: nextKeyValue re-fetches the spliced batch per row") { - // The per-emit ColumnarBatch is replaced on every nextBatch(), so a consumer holding on to an - // earlier getCurrentValue() would read the wrong vectors. Drives the non-columnar contract with + // One ColumnarBatch is handed out for the whole read and its key slots are rewritten per + // batch, so a consumer holding on to an earlier getCurrentValue() would read the wrong + // vectors. Drives the non-columnar contract with // a capacity small enough to span several batches. withTempDir { dir => val rows = (1L to 200L).map(i => (i, s"v_$i")) @@ -1482,14 +1669,14 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { assert(collected == expected, s"expected ${expected.size} rows across both schemas; got ${collected.size}") - // The avoided-bytes counters are what walk the offset index of the missing column. Their - // being populated and non-negative is the evidence that the walk ran and coped. + // The avoided-bytes counters are what walk the offset index of the missing column, so a + // positive total is the evidence that the walk ran and coped. Asserting non-negativity + // would prove nothing: `SQLMetric.add` drops a negative and the value cannot go below zero. val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) val bytesPf = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING) - assert(bytesRg.value >= 0 && bytesPf.value >= 0, - s"avoided-byte metrics must never go negative; got rg=${bytesRg.value} " + - s"pf=${bytesPf.value}") + assert(bytesRg.value + bytesPf.value > 0, + s"the walk must credit some avoided bytes; got rg=${bytesRg.value} pf=${bytesPf.value}") } } } @@ -1694,23 +1881,38 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val bloom = BloomFilterMightContain( Literal.create(null, BinaryType), XxHash64(Seq(AttributeReference("k", LongType)()), 42L)) - assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported") - assert(!subclass.supportsStorageFilter(bloom), "a subclass must not claim support") - // Not a bloom at all, and a bloom on a type the value copier has no branch for. - assert(!format.supportsStorageFilter(Literal.TrueLiteral)) val onVariant = BloomFilterMightContain( Literal.create(null, BinaryType), XxHash64(Seq(AttributeReference("v", VariantType)()), 42L)) - assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf") - // A key the reader could not evaluate on rows the plan would have excluded. In ANSI mode the - // cast throws on a string that is not a number, and the reader evaluates the predicate on every - // row the pushed filter left, including those an earlier conjunct would have dropped. + // A cast key is supported. In ANSI mode it can throw on a row an earlier conjunct would have + // dropped, and the reader handles that where it arises, by giving the filter up for the row + // group. Declining here instead would also decline every widening cast, which is what type + // coercion inserts for a join between an int and a bigint column. val onCast = BloomFilterMightContain( Literal.create(null, BinaryType), XxHash64(Seq(Cast(AttributeReference("s", StringType)(), LongType)), 42L)) - assert(!format.supportsStorageFilter(onCast), "a cast key can throw on rows the plan excluded") - // And the default is no support at all. - assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom)) + // What the reader cannot evaluate at all: a non-deterministic value side needs the + // `initialize(partitionIndex)` it never calls. + val onRand = BloomFilterMightContain( + Literal.create(null, BinaryType), + XxHash64(Seq(Cast(Rand(42L), LongType)), 42L)) + + // The conf is read in the format rather than in the planner, and it is the per-scan question + // that carries it, asked once before anything per conjunct. + assert(!format.supportsStorageFilterPushdown(spark), "the feature is off with the conf off") + withSQLConf(SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true") { + assert(format.supportsStorageFilterPushdown(spark), "and on with it on") + assert(!subclass.supportsStorageFilterPushdown(spark), "a subclass must not claim support") + assert(format.supportsStorageFilter(bloom), "a plain bloom on a long key is supported") + // Not a bloom at all, and a bloom on a type the value copier has no branch for. + assert(!format.supportsStorageFilter(Literal.TrueLiteral)) + assert(!format.supportsStorageFilter(onVariant), "VariantType has no primitive Parquet leaf") + assert(format.supportsStorageFilter(onCast), + "a cast key is pushed, and an evaluation error gives the row group up") + assert(!format.supportsStorageFilter(onRand), "a non-deterministic key cannot be evaluated") + // And the default is no support at all. + assert(!new NoStorageFilterFileFormat().supportsStorageFilter(bloom)) + } } @@ -1751,20 +1953,18 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("a file format without storage-filter support ignores a non-empty storageFilters") { - // The default `FileFormat.buildReaderWithStorageFilters` body may ignore what it is handed, - // because the post-scan Filter still holds it. Unreachable in production, since the planner - // asks `supportsStorageFilter` first. + test("a file format without storage-filter support declines rather than delegating") { + // The default `FileFormat.buildReaderWithStorageFilters` answers None, and the caller falls + // back to the ordinary builder. That is what keeps the two builders from being able to call + // each other, and declining is safe because the post-scan Filter still holds the conjunct. val storageFilters = Seq(GreaterThanOrEqual(BoundReference(0, LongType, nullable = false), Literal(1L))) - val e = intercept[Exception] { - new NoStorageFilterFileFormat().buildReaderWithStorageFilters( - spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters, - Map.empty, new Configuration()) - } - // This stub implements no reader at all, so the error it fails with is the delegation's, which - // is what shows the filters were not rejected. - assert(e.getMessage.contains("buildReader is not supported"), e.getMessage) + val declined = new NoStorageFilterFileFormat().buildReaderWithStorageFilters( + spark, new StructType(), new StructType(), new StructType(), Nil, storageFilters, + Map.empty, new Configuration()) + assert(declined.isEmpty, "the default must not build a reader of its own") + assert(!new NoStorageFilterFileFormat().supportsStorageFilterPushdown(spark), + "and it must not claim support either") } Seq(false, true).foreach { aqe => @@ -1777,16 +1977,23 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key -> "1000", SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "200", SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) - def run(pushdown: Boolean): Set[(Long, Long)] = withSQLConf( + def run(pushdown: Boolean): (Int, Set[(Long, Long)]) = withSQLConf( (baseConf + (SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> pushdown.toString)).toSeq: _* ) { - runBloomFilterJoin()._2.map(r => (r.getLong(0), r.getLong(1))).toSet + val (plan, rows) = runBloomFilterJoin() + (countBloomFiltersInStorageFilters(plan), + rows.map(r => (r.getLong(0), r.getLong(1))).toSet) } - val off = run(false) - val on = run(true) + val (offFilters, off) = run(false) + val (onFilters, on) = run(true) assert(on == off, s"results differ between conf-on and conf-off: on=$on off=$off") assert(on.nonEmpty, "the join should return rows, otherwise this proves nothing") + // Equal results are guaranteed by the post-scan Filter whether or not the scan applied the + // filter, so they alone would not notice AQE dropping it on the way to the final plan. + assert(offFilters == 0 && onFilters == 1, + s"the scan must carry the filter with the conf on and not with it off; " + + s"got on=$onFilters off=$offFilters") } } } @@ -1904,7 +2111,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { } } - test("page-subset ranges keep the avoided-byte metrics non-negative") { + test("page-subset ranges still credit the avoided bytes") { // The strict-subset branch of compressedBytesForRowRanges (offset index + dictionary page) only // runs when pushedFilterRanges is narrower than the block, which needs a multi-page row group. withTempDir { dir => @@ -1923,8 +2130,11 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val bytesRg = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP) val bytesPf = withSF.metrics(FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING) - assert(bytesRg.value >= 0 && bytesPf.value >= 0, - s"avoided-byte metrics must never go negative; rg=${bytesRg.value} pf=${bytesPf.value}") + // Non-negativity is not assertable: `SQLMetric.add` drops a negative, so the value cannot + // go below zero however wrong the arithmetic is. A positive total can fail. + assert(bytesRg.value + bytesPf.value > 0, + s"page-subset ranges must still credit avoided bytes; rg=${bytesRg.value} " + + s"pf=${bytesPf.value}") } } } @@ -2012,6 +2222,7 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession { val readerFn = new ParquetFileFormat().buildReaderWithStorageFilters( spark, schema, new StructType(), schema, pushedFilters, storageFilters, Map(FileFormat.OPTION_RETURNING_BATCH -> "true"), hadoopConf, metrics) + .getOrElse(fail("ParquetFileFormat must answer with a reader")) val file = PartitionedFile( InternalRow.empty, SparkPath.fromUrlString(s"${CountingLocalFileSystem.scheme}://$path"), From 2d8d39b68f9c0a470fb2fd04f0c0c99b60e497b0 Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Fri, 25 Sep 2026 19:02:10 +0200 Subject: [PATCH 5/6] Turn late materialization off when column-index filtering is disabled parquet.filter.columnindex.enabled=false says the file's page index is not to be trusted. Phase 0 honoured it, but phase 2 reads part of a row group through the offset index whatever the conf says, so a wrong index could pair a row's key with another row's values. The reader now declines the filter outright, which also removes the phase-0 branch and its helper. --- .../apache/spark/sql/internal/SQLConf.scala | 4 +- .../VectorizedParquetRecordReader.java | 40 +++++--------- .../parquet/ParquetStorageFilterSuite.scala | 54 ++++++++++--------- 3 files changed, 44 insertions(+), 54 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 06646331de02f..4ee6596fbadd2 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -1960,7 +1960,9 @@ object SQLConf { "more read of the key columns, since the phase that evaluated the filter already read " + "them. A file written with no Parquet page index is read with the filter applied only " + "where it empties a whole row group, since narrowing to part of one needs that index, so " + - "every row group of it that keeps a row pays that. Note that " + + "every row group of it that keeps a row pays that. Setting " + + "parquet.filter.columnindex.enabled to false turns this off entirely, because reading " + + "part of a row group goes through the page index. Note that " + "the surviving key values of a whole row group are buffered before the " + "first batch of that row group is produced, so a task holds up to one extra copy of the " + "key columns for one row group.") diff --git a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java index 679da1cffc2f8..f53801ba7f70d 100644 --- a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java +++ b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java @@ -246,12 +246,6 @@ public class VectorizedParquetRecordReader extends SpecificParquetRecordReaderBa private boolean[] keyRequired; private WritableColumnVector[] keyScratchVectors; private ColumnarBatch keyScratchBatch; - /** - * Mirrors parquet's {@code ParquetReadOptions.useColumnIndexFilter()}. Phase 0 consults it - * because {@link ParquetFileReader#getRowRanges(int)} does not: it checks only whether a filter - * is pushed, so it would keep narrowing by column index after a user disabled that filtering. - */ - private boolean useColumnIndexFilter = true; /** * Splicing state. Phase 1 keeps the surviving key values it has already decoded, and the emit @@ -715,6 +709,15 @@ public void setStorageFilter(ParquetStorageFilter storageFilter) { } private void initializeLateMaterialization() throws IOException { + if (!configuration.getBoolean(ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true)) { + // That conf is the escape hatch for a file whose page index is wrong, so it has to turn this + // feature off whole rather than only its filtering. Phase 2 reads part of a row group through + // the offset index, which parquet consults whatever the conf says, and a wrong one there + // pairs a row's key with another row's values. The post-scan filter cannot catch that: the + // key it sees is the right one. + storageFilter = null; + return; + } lateMatReader = reader.getUnderlyingReader(); if (lateMatReader == null) { // Late materialization drives a ParquetFileReader directly, so without one the filter is @@ -723,8 +726,6 @@ private void initializeLateMaterialization() throws IOException { storageFilter = null; return; } - useColumnIndexFilter = configuration.getBoolean( - ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED, true); // Resolve each key column's top-level ParquetColumn. Partition into present and missing (the // latter can happen under schema evolution: a column is in the requested schema but not in this // physical parquet file). For any non-primitive key we still bail; phase-1 reads only primitive @@ -969,8 +970,8 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { int blockIdx = nextBlockIndex++; long blockRowCount = lateMatReader.getRowGroups().get(blockIdx).getRowCount(); if (blockRowCount == 0) { - // parquet-mr never writes these, but RowRanges.createSingle(0) would build Range(0, -1) and - // trip parquet's own `from <= to` assertion. The plain read path skips them too. + // parquet-mr never writes these, but an empty block makes parquet's own getRowRanges build + // Range(0, -1) and trip its `from <= to` assertion. The plain read path skips them too. continue; } // Splicing buffers one key value per surviving row of the whole row group before it can emit @@ -987,7 +988,7 @@ private void loadNextRowGroupWithLateMaterialization() throws IOException { // requestedSchema goes back on first, because phases 1 and 2 narrow it and // ParquetFileReader.getRowRanges computes ranges against the reader's current paths. lateMatReader.setRequestedSchema(requestedColumns); - RowRanges pushedFilterRanges = pushedFilterRangesFor(blockIdx, blockRowCount); + RowRanges pushedFilterRanges = lateMatReader.getRowRanges(blockIdx); // RowRanges.rowCount() walks every range, so resolve each range set's count once. long baselineRows = pushedFilterRanges.rowCount(); if (baselineRows == 0) { @@ -1181,7 +1182,7 @@ private void recordFileSkipped() { // `getFilteredRecordCount()` at initialize resolved every block's ranges then. long blockRowCount = blocks.get(blockIdx).getRowCount(); if (blockRowCount == 0) continue; - RowRanges blockRanges = pushedFilterRangesFor(blockIdx, blockRowCount); + RowRanges blockRanges = lateMatReader.getRowRanges(blockIdx); long survivingRows = blockRanges.rowCount(); if (survivingRows == 0) continue; // The key columns are missing from this file, so they contribute nothing to the walk, and the @@ -1194,21 +1195,6 @@ private void recordFileSkipped() { } } - /** - * The rows of a block the pushed data filter allows, at column-index granularity. - * - * {@code getRowRanges} checks only whether a filter is pushed, not - * {@code options.useColumnIndexFilter()}, so calling it unconditionally would keep applying - * column-index filtering after a user turned it off, which is the escape hatch for a file whose - * column index is wrong. Every phase reads within these ranges, so a wrong column index would - * cost rows a plain read would have returned. - */ - private RowRanges pushedFilterRangesFor(int blockIdx, long blockRowCount) { - return useColumnIndexFilter - ? lateMatReader.getRowRanges(blockIdx) - : RowRanges.createSingle(blockRowCount); - } - /** * The block's column chunks by path, built once per row group and shared by the byte-metric calls * that consume it, since {@link BlockMetaData} offers no lookup of its own. diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala index 943104fd2095f..066f4cf4aa88c 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetStorageFilterSuite.scala @@ -2139,23 +2139,22 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession } } - test("column-index filtering off: phase 0 takes the whole row group") { - // Phase 0 asks parquet for row ranges only when column-index filtering is on, because - // ParquetFileReader.getRowRanges checks whether a filter is pushed and NOT whether the user - // enabled the column index. That branch is the escape hatch for a file whose column index is - // wrong, and nothing exercised it. Trusting a wrong one drops rows for good: every phase reads - // within phase 0's ranges, and no filter above the scan can bring back a row it never read. + test("column-index filtering off: the reader does not apply the filter at all") { + // `parquet.filter.columnindex.enabled=false` is the escape hatch for a file whose page index + // is wrong, and it has to cover this feature whole. Phase 0 could honour it on its own, but + // phase 2 reads part of a row group through the offset index, which parquet consults whatever + // that conf says, so a wrong index there would pair a row's key with another row's values. The + // post-scan filter cannot catch that, since the key it sees is the right one. // - // The row accounting is what tells the two arms apart. Everything is scoped to the rows the - // pushed data filter left, so with the column index on, the rows it prunes at page level never - // reach phase 1 and are never counted. With it off, every row of the block does, so emitted - // plus excluded covers the whole file. One row group with many pages keeps statistics-level - // row-group filtering out of it, which happens either way. + // The rows and the metrics together tell the two arms apart. This test executes the scan on + // its own, so nothing re-applies the conjunct above it: with the conf off the scan hands back + // every row of the file and reports nothing at all, not merely fewer skips. One row group with + // many pages keeps statistics-level row-group filtering out of it, which happens either way. withTempDir { dir => val rows = (1L to 400L).map(i => (i, f"v_$i%04d")) val path = writeParquetFile(dir, rows, rowGroupSize = 64 * 1024L, pageSize = Some(512L)) - def run(columnIndex: Boolean): (Set[(Long, String)], Long) = withSQLConf( + def run(columnIndex: Boolean): (Set[(Long, String)], Seq[Long]) = withSQLConf( SQLConf.PARQUET_STORAGE_FILTER_PUSHDOWN_ENABLED.key -> "true", ParquetInputFormat.COLUMN_INDEX_FILTERING_ENABLED -> columnIndex.toString) { val df = spark.read.parquet(path).select("k", "v").filter("k >= 350") @@ -2167,25 +2166,28 @@ class ParquetStorageFilterSuite extends QueryTest with SharedSparkSession val keyAttr = scan.output.find(_.name == "k").get val withSF = scan.copy(storageFilters = Seq(GreaterThanOrEqual(keyAttr, Literal(350L)))) val collected = executePlanCollect(withSF).toSet - val accounted = collected.size + - withSF.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP).value + - withSF.metrics(FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP).value - (collected, accounted) + val reported = Seq( + FileSourceScanLike.STORAGE_FILTER_ROW_GROUPS_SKIPPED, + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_BY_ROW_GROUP, + FileSourceScanLike.STORAGE_FILTER_ROWS_EXCLUDED_WITHIN_ROW_GROUP, + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_ROW_GROUP, + FileSourceScanLike.STORAGE_FILTER_BYTES_AVOIDED_BY_PAGE_FILTERING) + .map(withSF.metrics(_).value) + (collected, reported) } val expected = rows.filter(_._1 >= 350L).toSet - val (rowsOff, accountedOff) = run(columnIndex = false) - val (rowsOn, accountedOn) = run(columnIndex = true) - assert(rowsOff == expected, - s"with the column index off, got ${rowsOff.size} rows; expected ${expected.size}") + val (rowsOff, reportedOff) = run(columnIndex = false) + val (rowsOn, reportedOn) = run(columnIndex = true) + assert(rowsOff == rows.toSet, + s"with the column index off nothing must be filtered; got ${rowsOff.size} rows " + + s"of ${rows.size}") assert(rowsOn == expected, s"with the column index on, got ${rowsOn.size} rows; expected ${expected.size}") - assert(accountedOff == rows.size, - s"with the column index off every row of the file must be emitted or excluded; " + - s"accounted $accountedOff of ${rows.size}") - assert(accountedOn < rows.size, - s"with the column index on the pruned pages must not reach phase 1; " + - s"accounted $accountedOn of ${rows.size}") + assert(reportedOff.forall(_ == 0L), + s"with the column index off the filter must not run: $reportedOff") + assert(reportedOn.exists(_ > 0L), + s"with the column index on the filter must run, or this test proves nothing: $reportedOn") } } From 43c328c6c66a0fc4db2d4a231895d89157875554 Mon Sep 17 00:00:00 2001 From: Peter Toth Date: Fri, 25 Sep 2026 19:09:08 +0200 Subject: [PATCH 6/6] Document that supportsStorageFilter sees the original expression --- .../apache/spark/sql/execution/datasources/FileFormat.scala | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala index c7c206e727d22..0b01323f37996 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala @@ -220,6 +220,9 @@ trait FileFormat { * shapes and column types a reader supports stay in that reader's own package. Answering true * says the reader can evaluate the expression, not that it will: the conjunct stays in the * post-scan `Filter`, so a reader is free to give a file up. + * + * `expr` is the expression [[buildReaderWithStorageFilters]] will be given, not a canonicalized + * form of it, so a format may decide by column name or field metadata. */ def supportsStorageFilter(expr: Expression): Boolean = false