Skip to content

fix: avoid full listings for cached pruned partitions#23402

Open
kumarUjjawal wants to merge 1 commit into
apache:mainfrom
kumarUjjawal:fix/pruned-partition
Open

fix: avoid full listings for cached pruned partitions#23402
kumarUjjawal wants to merge 1 commit into
apache:mainfrom
kumarUjjawal:fix/pruned-partition

Conversation

@kumarUjjawal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The list files cache currently performs a full table listing on a cold cache miss, even when partition pruning has already narrowed the scan to a specific path prefix.

This makes selective partition queries pay the cost of listing the full table unless users disable the cache.

What changes are included in this PR?

  • List only the requested prefix on cold list-cache misses.
  • Keep using full-table cache entries to serve later prefix queries.
  • Ensure prefix-scoped cache entries do not satisfy full-table lookups.
  • Invalidate matching prefix-scoped list-cache entries on table writes.
  • Add regression tests that check the actual object store list prefix.

Are these changes tested?

Yes

Are there any user-facing changes?

Partition-pruned listing table queries with the list files cache enabled avoid full table listings on cold cache misses.
No API Change

@github-actions github-actions Bot added catalog Related to the catalog crate datasource Changes to the datasource crate labels Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalog Related to the catalog crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

datasource listing performs full listing despite user pruning partitions

1 participant