[format] Validate the avro block size before creating the writer - #9961
Merged
Merged
Conversation
AvroFileFormat passed file.block-size straight to Avro's setSyncInterval, which only accepts 32 bytes to 1 GiB. Other values failed at write time on the executor with Avro's own "Invalid syncInterval value: N" or, above 2 GiB, with "integer overflow" from Math.toIntExact, neither naming the option. Check the range up front and fail with the option name, value and bounds. The unchecked value also made DeletionVectorTest "select with format filter push down" flaky: it draws a random block size from [1, 10240] and a random format, so about one run in a thousand lands on avro with a block size below 32. It failed the Spark 3 / Scala 2.13 job of apache#9953 with block size 6 on a change unrelated to Avro. Draw from [32, 10271] instead. testFileBlockSizeOverflow, which pinned the raw ArithmeticException, is replaced by testFileBlockSizeOutOfAvroRange (1, 6, 31, 2^30+1, 2^31, 4 GiB+, Long.MAX_VALUE) and testFileBlockSizeAtAvroRangeBounds (32, 33, 2^30).
Contributor
|
+1 |
XiaoHongbo-Hope
pushed a commit
that referenced
this pull request
Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
AvroFileFormat.createBlockWriterpassesfile.block-sizestraight to Avro'sDataFileWriter.setSyncInterval, which only accepts 32 bytes to 1 GiB. Any other value fails at write time on the executor with Avro's own message (Invalid syncInterval value: 6) or, above 2 GiB, withArithmeticException: integer overflowfromMath.toIntExact— neither names the option.This also makes
DeletionVectorTest."Paimon deletionVector: select with format filter push down"flaky: it picks a random format andRandom.nextInt(10240) + 1as the block size, so about 1 run in 1,000 lands on avro with a block size below 32 and fails with the message above. It hit theJava / Spark 3 / Scala 2.13job of #9953 (https://github.com/apache/paimon/actions/runs/35309357679/job/105488023917, block size 6) on a change that does not touch Avro or deletion vectors; every other job passed. A CI failure like this sends a reviewer to look for a regression that is not there, so it is worth removing rather than re-running around.Change
AvroFileFormat: checkfile.block-sizeagainst[32 bytes, 1 GiB]before callingsetSyncInterval, and fail withfile.block-size for avro must be between 32 bytes and 1 gb, but was N bytes.The check also covers the former integer-overflow case. Reads are unaffected; the check runs only when a writer is created.DeletionVectorTest: draw the random block size from[32, 10271]instead of[1, 10240].Tests
AvroFileFormatTest:testFileBlockSizeOutOfAvroRange(7 values: 1, 6, 31, 2^30+1, 2^31, 4 GiB+,Long.MAX_VALUE) asserts anIllegalArgumentExceptionnamingfile.block-size, the value and the bounds. On master these fail withInvalid syncInterval value: N(first four) orinteger overflow(last three); it replacestestFileBlockSizeOverflow, which pinned the raw overflow.testFileBlockSizeAtAvroRangeBounds(32, 33, 2^30) writes successfully.testFileBlockSizecases (no block size, 1 kb … 128 kb) andtestManifestIgnoresDataFileBlockSizeare unchanged and pass.Results:
AvroFileFormatTest55/55,org.apache.paimon.format.avro.*79/79, SparkDeletionVectorTest17/17 (Spark 3, Scala 2.12, shaded build), checkstyle and spotless clean.API and Format
No API or format changes. Behaviour change: an out-of-range avro
file.block-sizenow fails with a message naming the option instead of an Avro-internal message or anArithmeticException; it still fails at the same point (writer creation).