Skip to content

[core] Prune manifest blocks with sidecars during scans - #9897

Open
leaves12138 wants to merge 4 commits into
apache:masterfrom
leaves12138:codex/manifest-sidecar-read
Open

leaves12138 wants to merge 4 commits into
apache:masterfrom
leaves12138:codex/manifest-sidecar-read

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Wire manifest sidecars into Java scans, following the format (#9845), lifecycle (#9886), and writer (#9889) changes. Extracted from #9743 after rebasing the tracking PR onto current master.

  • Reuse manifest.sidecar.enabled; no additional read option is introduced.
  • Select physical Avro blocks using partition, row-ID and conservative bucket predicates before reading manifest entries. Empty selections skip the manifest body entirely.
  • Preserve normal entry filters, statistics filtering, conversion and ADD/DELETE reconciliation.
  • Follow only explicit sidecar references. Missing, corrupt or unsupported sidecars fall back to ordinary manifest reads. Disabled sidecars and scans without relevant filters skip sidecar access.
  • Share the manifest byte cache with sidecar and selected-block reads, keeping block keys separate from whole-manifest entry keys. Partial results cannot populate the whole-manifest entry cache.
  • Preserve the existing ManifestFile.read overloads for callers that do not supply a block selection.

No file-format, writer, lifecycle or Python changes are included.

Tests

Java 8: 195 targeted tests passed with Checkstyle and Spotless enabled.

Tests cover row-ID, partition-only and bucket-only pruning; nullable coverage; real seek/byte-read accounting; repeated partial/all-block cache reads; explicit/custom references; disabled and unfiltered reads; missing/corrupt sidecar fallback while preserving deletes and overlapping row-ID groups; and existing data-evolution, bucket-filter, primary-key and global-index scans.

mvn -B -ntp -nsu -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false -Dflink.forkCount=2 \
  -Dtest=ManifestFileTest,ManifestSidecarTest,ManifestSidecarWriteTest,ManifestBlockIndexTest,DataEvolutionFileStoreScanTest,BucketFilterScanTest,KeyValueFileStoreScanTest,DataEvolutionBatchScanTest test

@leaves12138
leaves12138 marked this pull request as ready for review September 16, 2026 14:15
@leaves12138 leaves12138 reopened this Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant