Skip to content

[core] Unify manifest entry caching by physical block - #9919

Closed
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/manifest-block-entry-cache
Closed

leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:codex/manifest-block-entry-cache

Conversation

@leaves12138

Copy link
Copy Markdown
Contributor

Purpose

Unify the whole-manifest entry cache and the compressed Avro block cache used by sidecar-selected reads into one decoded-entry cache at physical block granularity.

This draft is based on current master, after #9897 and #9908. It contains only the Java cache changes and their tests/documentation, not the deferred Python writer integration from #9743.

Changes

  • Change ManifestEntryCache to ObjectsCache<BlockKey, ManifestEntry, ManifestEntrySegments>, keyed by manifest path, offset and encoded block length.
  • Let full reads and sidecar-selected reads reuse complete, unfiltered decoded blocks. Apply each query's predicates and converter after retrieving the block.
  • Remove the separate compressed-block cache from SelectedBlockInput; keep the independent sidecar metadata-byte cache unchanged.
  • Cache a complete physical block directory after reaching EOF or selecting all blocks, so a subsequent full read can find previously cached entries.
  • Preserve physical entry order and ADD/DELETE semantics. Account for decoded buffers and lookup metadata, and replay an oversized block without retaining a partial cache entry or re-reading the file.
  • Add coverage for full/selected cache reuse, distinct keys, filters, ordering, failure paths, memory limits and concurrent queries.

Scope

No manifest/sidecar on-disk format or writer behavior changes. The cache.manifest-sidecar.max-memory option and its default are unchanged. This is a draft for review of the cache design and implementation; production performance validation is not claimed here.

Validation

  • Formatting and git diff --check passed.
  • Targeted Maven reactor tests passed on the rebased branch: core 369 tests (1 skipped), common 2 tests, with no failures or errors.
  • Command: mvn -pl paimon-core -am -DwildcardSuites=none -Dtest=Manifest*Test,ObjectsCacheTest,SegmentsCacheTest,DataPagedOutputSerializerTest,DataEvolutionBatchScanTest,CoreOptionsTest -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test.

@JingsongLi JingsongLi closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants