Memory budget excludes hydrate/squeeze transients: tier reports at limit while the process holds 3x that #43

Summary

The memory budget accounts for an entry only once it has landed in the index. Every
intermediate that the hydrate → insert → squeeze cycle creates on the way there is
invisible to it — and that cycle runs per batch for any scan whose working set exceeds
the memory tier.

Measured on a 16 GiB container with a 5.15 GB memory tier and a 30 GB disk tier, scanning
a synthetic table of ~960k rows at ~7.5 KB/row (~7.3 GB): the cache's own
used_memory_bytes sat at 5.14 GB against its 5.15 GB limit — perfect by its own
bookkeeping — while process anonymous memory reached 16.35 GB and the container was
OOM-killed. At rest, with no queries running, the same process holds 8.64 GB against a
tally of 5.04 GB.

Because the host query engine's memory pool only tracks its own operators, nothing in the
system can see or refuse these bytes.

Where the accounting stops

All line references are at rev 3aec144.

• BudgetAccounting::try_reserve_memory (src/core/src/cache/budget.rs:38) has exactly
  one non-test reserving caller: CacheStorage::try_insert
  (src/core/src/cache/core.rs:404-425), which reserves at the moment the entry is
  inserted into self.index. (The release path,
  drop_memory_entry → try_update_memory_usage(…, 0) at core.rs:451, is the other
  budget touchpoint.)
• read_disk_liquid_array (src/core/src/cache/core.rs:840-861) fetches the object and
  runs ipc::read_from_bytes into a LiquidArrayRef with no budget interaction at
  all.
• maybe_hydrate (defined src/core/src/cache/core.rs:569; read-path call sites at 160,
  651, 673, 766, 894, 916) then promotes that decoded entry toward the memory tier.
• insert_inner (src/core/src/cache/core.rs:339-369) loops: try_insert → on failure
  find_memory_victim(8) → squeeze_victims → retry, holding the pending entry across
  every iteration.

So each iteration can have all of the following live and uncounted simultaneously:

1. the decoded disk-tier entry,
2. the hydrated copy being inserted,
3. up to 8 victims' transcode outputs (TranscodeSqueezeEvict compresses to FSST in
   memory before the result is indexed),
4. the Arrow array materialized for the caller.

Reading from the cache triggers this, not just filling it

This is what makes it a continuous cost rather than a warm-up cost, and it was active
during the OOM itself.

squeeze is reached from both read_parquet_batch_and_fill_cache and
CachedColumn::get_arrow_array_with_filter. In the heap profile of the window that ended
in the kill, the read-side path is:

CachedColumn::get_arrow_array_with_filter        21.4%
  ├─ LiquidCache::read_disk_liquid_array         20.7%
  └─ futures_util::stream::collect::Collect::poll
       └─ TranscodeSqueezeEvict::squeeze         43.5%

That intermediate Collect node is the buffer_unordered(…).collect() inside
squeeze_victims, so the graph shows the read path entering the concurrent squeeze
directly. A disk-tier read re-inserts into the memory tier, the tier is full, and the
insert loop squeezes 8 victims.

It follows from AlwaysHydrate being the default hydration policy
(src/datafusion-local/src/lib.rs:91): a query whose working set is larger than the memory
tier churns indefinitely, warm or cold. A warm profile makes it unambiguous only because
there are no parquet-decode frames left to confuse it with — 49.2% of live heap is FSST
transcode with zero decode frames present.

Two aggravating details

Squeeze concurrency is unbounded, for victims that write to disk. squeeze_victims
(src/core/src/cache/core.rs:493-510) uses buffer_unordered(usize::MAX) with no byte
bound, so nothing limits how many victims' outputs are resident together.

The scope needs stating precisely, because an earlier revision of this issue overstated
it. These are futures on the calling task, so they only overlap across an await. A
MemoryArrow victim whose transcode succeeds returns bytes_to_write: None
(src/core/src/cache/policies/squeeze.rs:116-119) — no disk write, hence no await, hence
each such future runs to completion in a single poll and no two are ever mid-squeeze
together. The pile-up requires a victim already in liquid form, whose squeeze writes
its backing to disk and therefore awaits. Measured on that case: peak in-flight reaches
about 9x one entry's size (all eight victims plus the pending entry) unbounded, against
~3x when byte-bounded.

Production takes the concurrent path; the tests only take the sequential one.
src/core/src/cache/builders.rs:64 sets squeeze_victims_concurrently: !cfg!(test). So
the memory shape that ships is the one no test exercises.

Transcode buffers grow by doubling. 69.6% of sampled bytes in that window arrive via
do_rallocx (realloc growth) rather than fresh allocation, which both leaves doubling
slack in finished buffers and frees a long tail of half-grown ones.

Measurements

Profile shares are from inuse_space with lg_prof_sample:20; attribution is by
allocation site, not by holder. Note that a shallow node cut hides this mechanism: at a
top-28 cut the squeeze frame falls below the threshold and its incoming read-path edge
collapses into the truncation node, which is what made an earlier revision of this issue
claim the read-triggered squeeze was absent when cold. It is not.

Suggested directions

1. Count in-flight bytes against the budget, not just indexed ones — reserve before
   decoding/hydrating/transcoding and release on failure, so the tier's limit bounds the
   cycle's true footprint rather than its end state.
2. Make hydration conditional. Skip re-inserting a disk-tier hit when the caller's
   working set clearly exceeds the memory tier. A policy that always hydrates converts
   every read of an oversized working set into an eviction storm.
3. Bound squeeze concurrency — buffer_unordered with a small constant instead of
   usize::MAX.
4. Expose an in-flight gauge alongside used_memory_bytes. The current gauge reading
   exactly at limit while the process holds three times that is the reason this went
   unnoticed.
5. Cover the concurrent squeeze path in tests, given it is the only one that ships.

Not established

• The idle excess is unexplained. 8.64 GB resident against a 5.04 GB tally with nothing
  running. CachedBatch::memory_usage_bytes
  (src/core/src/cache/cached_batch.rs:68-75) delegates to get_array_memory_size, and
  LiquidByteViewArray::get_detailed_memory_usage
  (src/core/src/liquid_array/byte_view_array/mod.rs:225-239) sums its components
  including the FSST buffer's capacity — so simple slack does not obviously explain it.
  One candidate: reference_count (cached_batch.rs:78) is reported in stats but does not
  gate eviction, so an entry can be evicted — dropping its contribution to the tally to
  zero for a disk-tier entry — while an outstanding Arc keeps the array alive.
• The split between live intermediates and allocator-retained pages at peak. Only
  ~12.4 GB of the 16.35 GB peak is live heap. The remainder is plausibly freed-but-unpurged
  pages given the realloc churn, but a heap profile cannot attribute those. jemalloc's
  stats.allocated vs stats.resident would settle it; in our build nothing enables
  tikv-jemalloc-sys/stats, so jemalloc compiles --disable-stats and those counters do
  not exist. Comparing a single profile upload's total against process anon is the
  available substitute.