Add row_groups_per_file to to_parquet/write_parquet - #620
Open
joaquinhuigomez wants to merge 1 commit into
Open
joaquinhuigomez wants to merge 1 commit into
joaquinhuigomez wants to merge 1 commit into
Conversation
The Parquet COPY option ROW_GROUPS_PER_FILE is supported by the engine but was not exposed on DuckDBPyRelation.to_parquet/write_parquet, so callers had to drop down to raw SQL to rotate output files by row group count. Pass the value straight through to the COPY options map. Validation mirrors the neighbouring row_group_size block: non-integer input raises InvalidInputException rather than reaching the engine. Fixes duckdb#386
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The engine already supports the
ROW_GROUPS_PER_FILEparquet COPY option, but the Python binding never exposed it, so rotating output files by row-group count was only reachable through raw SQL (#386).This passes the argument through as a COPY option, mirroring the existing
row_group_sizehandling: int-only validation inToParquet, registered on both theto_parquetandwrite_parquetaliases, and added to the type stubs.Tests follow the engine's own
row_groups_per_file.testfixture (10k rows,row_group_size=2000,row_groups_per_file=1→ 5 files) on athreads=1connection, since rotation is best-effort under multi-threaded writes; a second test covers the non-int rejection. Before the change both methods raiseTypeErroron the kwarg while the equivalent SQL COPY works.tests/fast/apipasses (375 passed; the 2 new tests are the only delta vs. the pre-change baseline) and pre-commit (ruff, clang-format, mypy) is clean.One note:
row_groups_per_file=Trueis accepted becauseboolsubclassesint— same pre-existing behavior asrow_group_size, left untouched for consistency.Fixes #386