feat: report JVM Arrow allocations to Spark's memory manager - #5998

Draft

andygrove wants to merge 13 commits into

apache:mainfrom

andygrove:arrow-allocator-accounting

Commits on Sep 17, 2026

feat: report JVM Arrow allocations to Spark's memory manager
andygrove
committed
fda7bd6
refactor: reduce lock traffic in Arrow allocation accounting
andygrove
committed
0e3dfa7
fix: exclude FFI-imported buffers from Arrow allocation accounting
andygrove
committed
55903fd
docs: record that JVM Arrow allocations are now reported to Spark
andygrove
committed
2d531ec
style: keep the new config doc within the 100 character line limit
andygrove
committed
4d7657d
style: give the imported Arrow allocator an explicit result type
andygrove
committed
a2518d9
fix: bind JVM Arrow accounting to a per-task allocator
andygrove
committed
e58b177
test: cover release ownership, task completion and failed acquisitions
andygrove
committed
094863f
test: benchmark the cost of reporting Arrow allocations to Spark
andygrove
committed
d591b8f
fix: contain acquisition failures and stop double-charging FFI buffers
andygrove
committed
054fe9a

Commits on Sep 18, 2026