Preserve Parquet reader progress and add opt-in bounded range cache

datafusion_iceberg/AGENTS.md

@@ -0,0 +1,22 @@
+# datafusion_iceberg
+
+This module owns DataFusion providers and physical scans over Iceberg snapshots.
+Keep schema, delete-file and snapshot semantics intact when optimizing reads.
+
+`parquet_metadata_cache.rs` wraps every Iceberg Parquet reader with the shared
+footer cache. `parquet_data_cache.rs` optionally caches immutable byte ranges;
+the default capacity is zero. Both use store-qualified file identity and bounded
+memory. Keep data-cache metrics attached to each scan. Preserve batched I/O on
+misses, do not hold synchronous locks across awaits, and account for retained
+buffer ownership rather than only a shared slice's visible length.
+
+Validate cache changes with focused unit tests for identities, memory limits,
+partial hits, error recovery, and representative Iceberg query result checks.
+Consumers must include configured cache capacity in their memory budgeting.
+
+Metadata readers must remain independently pollable. Never await a per-file
+load gate owned by another query input: a prefetched probe can be paused while
+the build input needs the same footer. Contended cold loads currently read
+independently, expose a bypass counter and preserve the byte-capped warm cache.
+Any future miss coalescing must pass the paused-loader liveness regression and
+must not detach unbounded work or retain canceled readers in a global cache.

datafusion_iceberg/README.md

@@ -1,3 +1,40 @@
 # Datafusion iceberg
 
-Provides the functionality to use apache iceberg with datafusion including the `TableProvider`, `SchemaProvider` and `CatalogProvider` traits.
+Provides the functionality to use apache iceberg with datafusion including the `TableProvider`, `SchemaProvider` and `CatalogProvider` traits.
+
+## Parquet footer cache and pipeline progress
+
+The process-wide parsed-footer cache is byte-capped by
+`ICEBERG_PARQUET_METADATA_CACHE_MB` (default 64 MiB, zero disables it) and preserves
+store-qualified immutable-file identity plus a file-size sanity stamp. Warm
+reads reuse parsed metadata without object-store I/O.
+
+Cold readers do not wait for a per-file gate owned by another reader. A lazy
+query pipeline can stop polling a prefetched probe while a build input needs
+the same footer; blocking single-flight would then deadlock. A contended cold
+reader instead performs its own bounded read and records
+`parquet_metadata_cache_contention_bypasses`. Duplicate cold fetches are an
+intentional liveness trade, not a query-result cache or a background fill task.
+The retained cache remains byte-capped; concurrent reads and query-owned
+metadata require their own headroom. A deterministic paused-loader unit test
+covers this scheduling condition independently of benchmark table names.
+
+## Immutable Parquet data cache
+
+`ICEBERG_PARQUET_DATA_CACHE_MB` enables a process-wide, byte-bounded LRU for exact
+Parquet read ranges. Its default is `0` (disabled). Cache identity includes the
+store-qualified file URI, file size, and range. Iceberg rewrites and time travel
+use immutable data-file paths; query results, table snapshots and credentials
+are not cached here. Metadata resolution and delete-file handling still run.
+
+Warm hits avoid object-store reads while retaining normal Parquet decoding and
+DataFusion execution. Misses keep the reader's batched multi-range API. Retained
+bytes own compact buffers, so a small slice cannot pin a large coalesced response.
+Entries plus conservative key overhead count against the cap. The process's
+other memory users and in-flight reads still need their own budget.
+
+Scan metrics expose `parquet_data_cache_hits`, `parquet_data_cache_misses`,
+`parquet_data_cache_hit_bytes`, and `parquet_data_cache_miss_bytes` through
+`EXPLAIN ANALYZE VERBOSE`. This initial implementation does not reuse overlapping
+ranges or deduplicate concurrent misses. It is intended for immutable Iceberg
+files, not files overwritten in place at the same URI and size.

datafusion_iceberg/src/lib.rs

@@ -1,6 +1,7 @@
 pub mod catalog;
 pub mod error;
 pub mod materialized_view;
+mod parquet_data_cache;
 mod parquet_metadata_cache;
 pub mod planner;
 mod pruning_statistics;