Skip to content

[python] Read VARIANT pages with Parquet OffsetIndex - #9962

Merged
JingsongLi merged 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read
Sep 22, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/late-materialize-variant-read

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What

  • Extend the Parquet OffsetIndex row-window reader from [python] Read Parquet row windows with OffsetIndex #9850 to standard VARIANT columns.
  • Read only the pages needed by a contiguous row window when it is beneficial.
  • Keep the ordinary reader fallback for missing indexes, scattered ranges, and unprofitable selections.

I/O trade-off

Selective reads can transfer fewer bytes but may issue more object-store range requests. This change does not alter the table format or page-index option semantics.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add opt-in Parquet OffsetIndex reads for selected rows [python] Add opt-in late materialization using Parquet OffsetIndex Sep 18, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/late-materialize-variant-read branch 2 times, most recently from fc224a8 to 4846dba Compare September 21, 2026 09:47
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add opt-in late materialization using Parquet OffsetIndex [python] Read VARIANT pages with Parquet OffsetIndex Sep 21, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 21, 2026 11:24
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/late-materialize-variant-read branch from 4846dba to d22f1da Compare September 21, 2026 11:41

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: CLEAN in this focused pass. Removing the bounded-VARIANT exclusion lets a row-range read use Parquet OffsetIndex instead of decoding a full row group. The added tests compare selected-page output with the ordinary reader across page versions and null rows, measure fewer bytes read, and exercise the unprofitable and missing-index fallbacks. The production change is one guard removal; I found no changed behavior outside that path in the diff. I reviewed code and tests but did not run PyPaimon locally.

@JingsongLi
JingsongLi merged commit 8e68137 into apache:master Sep 22, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants