Skip to content

Preserve Parquet reader progress and add opt-in bounded range cache - #80

Merged
Vedin merged 2 commits into
embucket-sync-df55.0.0from
autoresearch/interactive-iceberg-integration
Sep 15, 2026
Merged

Vedin merged 2 commits into
embucket-sync-df55.0.0from
autoresearch/interactive-iceberg-integration

Conversation

@Vedin

@Vedin Vedin commented Sep 15, 2026

Copy link
Copy Markdown

Summary

  • Keep cold Parquet metadata readers independently pollable: a reader whose
    parent has paused it must not hold up another input reading the same file.
    Contention now performs an independent read rather than waiting on that
    reader's gate. Add a bypass metric and a deterministic paused-loader regression.
  • Add an optional byte-bounded immutable Parquet range cache, disabled by default
    (ICEBERG_PARQUET_DATA_CACHE_MB=0). Keys include store-qualified file identity,
    file size and exact range. Misses preserve batched I/O; retained ranges own
    compact buffers, with entry/key overhead included in the bound.
  • Document capacity, memory headroom, immutable-file assumptions, cancellation
    and the intentional duplicate-cold-I/O tradeoff. Preserve current row-lineage
    code and existing metadata cache defaults.

Verification

  • Fresh local cargo fmt --all -- --check: PASS.
  • Fresh local library Clippy with -D warnings: PASS on DataFusion 55.1.0 with
    the Rustice Arrow integration. Cargo metadata verifies a single 31-package
    DF55.1 closure (no mixed DF55.0/55.1 packages).
  • All 28 actual-module cache unit tests: PASS, zero ignored. The local test
    wrapper compiles the two real source files, while the library Clippy run checks
    their real reader-factory integration. The wrapper avoids unrelated bundled
    DuckDB/Python test dependencies; this is not a claim that full repository CI
    has run locally
    . The PR's full CI remains required.
  • Clippy of the actual cache modules including their tests, with -D warnings:
    PASS. The first CI run caught two single_range_in_vec_init lints in test
    setup; ranges are now constructed explicitly without changing assertions or
    allowing the lint. Runtime code is unchanged by this follow-up.
  • Source hashes and byte-for-byte publication/snapshot comparisons checked before
    and after the local gate (iceberg55-r9, Rust 1.95.0).

The selected implementation was exercised during the earlier E01–E21 campaign
on TPC-H SF100 and ClickBench100m, locally and on SPCS. Those are historical
end-to-end results, not an isolated cache speedup and not qualification of the
current-main integration. A 1 GiB range-cache trial regressed its screen, so the
range cache remains opt-in. Fresh integrated benchmarks are a separate Rustice
merge gate. No raw research logs or result workbooks are included in this PR.

Limits

The footer cache trades possible duplicate cold reads for progress; retained
metadata stays bounded, but in-flight/query-owned memory needs separate headroom.
The range cache does not merge overlapping ranges or deduplicate concurrent
misses. Same-URI/same-size mutable files are not supported by its identity model.
No query-result caching, new runtime dependencies, detached background work, or
test skips are introduced.

Port the selected immutable-cache changes onto current row-lineage main. Keep range caching disabled by default and cold metadata readers independently pollable. Fresh DF55.1 integration library Clippy, format, and all 28 actual-module cache tests pass; full repository CI remains required on the PR.
Preserve all assertions and runtime code. Rust 1.95 library and actual-module test Clippy with -D warnings, format, and all 28 cache tests pass on the resolved DF55.1 integration.
@Vedin
Vedin merged commit ee26132 into embucket-sync-df55.0.0 Sep 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant